[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790713339161492-17621-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 20:22:19 dist-test-slave-zbnc krb5kdc[17628](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 20:22:19 dist-test-slave-zbnc krb5kdc[17628](info): set up 2 sockets
Sep 29 20:22:19 dist-test-slave-zbnc krb5kdc[17628](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260929 20:22:20.300310 17621 mini_kdc.cc:121] Time spent starting KDC: real 1.052s user 0.002s sys 0.013s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790713339161492-17621-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790713339161492-17621-0/krb5kdc/test-user.keytab.
Sep 29 20:22:20 dist-test-slave-zbnc krb5kdc[17628](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713340, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T20:22:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T20:22:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.17.53.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.53.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.53.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.126.keytab.
Entry for principal kudu/127.17.53.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.126.keytab.
I20260929 20:22:20.475925 17621 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
/tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790713339161492-17621-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790713339161492-17621-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790713339161492-17621-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790713339161492-17621-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.53.126:40255
--webserver_interface=127.17.53.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.126.keytab
--principal=kudu/127.17.53.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.53.84:33983
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.53.126:40255
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790713339161492-17621-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790713339161492-17621-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790713339161492-17621-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:22:20.800463 17641 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:22:20.800778 17641 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:22:20.800936 17641 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:22:20.808151 17641 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:22:20.808291 17641 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:22:20.808398 17641 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:22:20.808487 17641 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:22:20.808564 17641 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:22:20.818681 17641 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.53.84:33983
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790713339161492-17621-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790713339161492-17621-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.53.126:40255
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.126.keytab
--principal=kudu/127.17.53.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.53.126:40255
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790713339161492-17621-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.53.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790713339161492-17621-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:22:20.820091 17641 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:22:20.821520 17641 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:22:20.833600 17646 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:22:20.833756 17649 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:22:20.833550 17647 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:22:20.834384 17641 server_base.cc:1034] running on GCE node
I20260929 20:22:20.835193 17641 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:22:20.836529 17641 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:22:20.837734 17641 hybrid_clock.cc:648] HybridClock initialized: now 1790713340837680 us; error 37 us; skew 500 ppm
I20260929 20:22:20.838099 17641 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:22:20 dist-test-slave-zbnc krb5kdc[17628](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713340, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.53.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:22:20.841253 17641 init.cc:377] Logged in from keytab as kudu/127.17.53.126@KRBTEST.COM (short username kudu)
I20260929 20:22:20.842629 17641 webserver.cc:460] Webserver started at http://127.17.53.126:36411/ using document root <none> and password file <none>
I20260929 20:22:20.843353 17641 fs_manager.cc:359] Metadata directory not provided
I20260929 20:22:20.843452 17641 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:22:20.843699 17641 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:22:20.846643 17641 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790713339161492-17621-0/minicluster-data/master-0/data/instance:
uuid: "c5587c5ecdc74dda810d4b3f5201053a"
format_stamp: "Formatted at 2026-09-29 20:22:20 on dist-test-slave-zbnc"
server_key: "64d49591d86f8832471a1f48e615e20e"
server_key_iv: "9dee9612ec0de4075dc5d4d537b22086"
server_key_version: "encryptionkey@0"
I20260929 20:22:20.847432 17641 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790713339161492-17621-0/minicluster-data/master-0/wal/instance:
uuid: "c5587c5ecdc74dda810d4b3f5201053a"
format_stamp: "Formatted at 2026-09-29 20:22:20 on dist-test-slave-zbnc"
server_key: "64d49591d86f8832471a1f48e615e20e"
server_key_iv: "9dee9612ec0de4075dc5d4d537b22086"
server_key_version: "encryptionkey@0"
I20260929 20:22:20.852661 17641 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 20:22:20.856130 17656 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:22:20.857508 17641 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 20:22:20.857693 17641 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790713339161492-17621-0/minicluster-data/master-0/data,/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790713339161492-17621-0/minicluster-data/master-0/wal
uuid: "c5587c5ecdc74dda810d4b3f5201053a"
format_stamp: "Formatted at 2026-09-29 20:22:20 on dist-test-slave-zbnc"
server_key: "64d49591d86f8832471a1f48e615e20e"
server_key_iv: "9dee9612ec0de4075dc5d4d537b22086"
server_key_version: "encryptionkey@0"
I20260929 20:22:20.857892 17641 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790713339161492-17621-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790713339161492-17621-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790713339161492-17621-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:22:20.890239 17641 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:22:20.896448 17641 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:22:20.896799 17641 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:22:20.919844 17641 rpc_server.cc:307] RPC server started. Bound to: 127.17.53.126:40255
I20260929 20:22:20.919839 17708 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.53.126:40255 every 8 connection(s)
I20260929 20:22:20.921980 17641 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790713339161492-17621-0/minicluster-data/master-0/data/info.pb
I20260929 20:22:20.923563 17621 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu as pid 17641
I20260929 20:22:20.923905 17621 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790713339161492-17621-0/minicluster-data/master-0/wal/instance
I20260929 20:22:20.926415 17621 external_mini_cluster.cc:1428] Setting key 4efebfbbf245a2186d303562cc3fc824
I20260929 20:22:20.927047 17709 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:22:20.938345 17709 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c5587c5ecdc74dda810d4b3f5201053a: Bootstrap starting.
I20260929 20:22:20.941764 17709 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c5587c5ecdc74dda810d4b3f5201053a: Neither blocks nor log segments found. Creating new log.
I20260929 20:22:20.943223 17709 log.cc:824] T 00000000000000000000000000000000 P c5587c5ecdc74dda810d4b3f5201053a: Log is configured to *not* fsync() on all Append() calls
I20260929 20:22:20.946168 17709 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c5587c5ecdc74dda810d4b3f5201053a: No bootstrap required, opened a new log
I20260929 20:22:20.954953 17709 raft_consensus.cc:348] T 00000000000000000000000000000000 P c5587c5ecdc74dda810d4b3f5201053a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5587c5ecdc74dda810d4b3f5201053a" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 40255 } }
I20260929 20:22:20.955343 17709 raft_consensus.cc:374] T 00000000000000000000000000000000 P c5587c5ecdc74dda810d4b3f5201053a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:22:20.955447 17709 raft_consensus.cc:729] T 00000000000000000000000000000000 P c5587c5ecdc74dda810d4b3f5201053a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5587c5ecdc74dda810d4b3f5201053a, State: Initialized, Role: FOLLOWER
I20260929 20:22:20.956075 17709 consensus_queue.cc:260] T 00000000000000000000000000000000 P c5587c5ecdc74dda810d4b3f5201053a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5587c5ecdc74dda810d4b3f5201053a" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 40255 } }
I20260929 20:22:20.956311 17709 raft_consensus.cc:388] T 00000000000000000000000000000000 P c5587c5ecdc74dda810d4b3f5201053a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 20:22:20.956439 17709 raft_consensus.cc:482] T 00000000000000000000000000000000 P c5587c5ecdc74dda810d4b3f5201053a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 20:22:20.956568 17709 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c5587c5ecdc74dda810d4b3f5201053a [term 0 FOLLOWER]: Advancing to term 1
Sep 29 20:22:20 dist-test-slave-zbnc krb5kdc[17628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713340, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.53.126@KRBTEST.COM
I20260929 20:22:20.959319 17709 raft_consensus.cc:504] T 00000000000000000000000000000000 P c5587c5ecdc74dda810d4b3f5201053a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5587c5ecdc74dda810d4b3f5201053a" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 40255 } }
I20260929 20:22:20.960016 17709 leader_election.cc:302] T 00000000000000000000000000000000 P c5587c5ecdc74dda810d4b3f5201053a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c5587c5ecdc74dda810d4b3f5201053a; no voters:
I20260929 20:22:20.960749 17709 leader_election.cc:288] T 00000000000000000000000000000000 P c5587c5ecdc74dda810d4b3f5201053a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 20:22:20.960870 17714 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c5587c5ecdc74dda810d4b3f5201053a [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:22:20.963457 17709 sys_catalog.cc:564] T 00000000000000000000000000000000 P c5587c5ecdc74dda810d4b3f5201053a [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:22:20.964171 17714 raft_consensus.cc:686] T 00000000000000000000000000000000 P c5587c5ecdc74dda810d4b3f5201053a [term 1 LEADER]: Becoming Leader. State: Replica: c5587c5ecdc74dda810d4b3f5201053a, State: Running, Role: LEADER
I20260929 20:22:20.964870 17714 consensus_queue.cc:237] T 00000000000000000000000000000000 P c5587c5ecdc74dda810d4b3f5201053a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5587c5ecdc74dda810d4b3f5201053a" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 40255 } }
I20260929 20:22:20.970073 17711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:20.931182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54526 (local address 127.17.53.126:40255)
0929 20:22:20.932165 (+ 983us) server_negotiation.cc:206] Beginning negotiation
0929 20:22:20.932181 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 20:22:20.933518 (+ 1337us) server_negotiation.cc:407] Connection header received
0929 20:22:20.942954 (+ 9436us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:22:20.942987 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:22:20.943484 (+ 497us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:22:20.943960 (+ 476us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:22:20.947159 (+ 3199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:20.949328 (+ 2169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:20.951087 (+ 1759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:20.952190 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:20.952380 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:22:20.958076 (+ 5696us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:22:20.958089 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:22:20.958111 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:22:20.958175 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:22:20.962440 (+ 4265us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:22:20.963579 (+ 1139us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:22:20.963590 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:22:20.963602 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:22:20.963820 (+ 218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:22:20.964494 (+ 674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:22:20.964498 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:22:20.964502 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:22:20.965353 (+ 851us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:22:20.968165 (+ 2812us) server_negotiation.cc:1035] Waiting for connection context
0929 20:22:20.968395 (+ 230us) server_negotiation.cc:299] Negotiation successful
0929 20:22:20.968691 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":724,"thread_start_us":472,"threads_started":1}
I20260929 20:22:20.971995 17715 sys_catalog.cc:455] T 00000000000000000000000000000000 P c5587c5ecdc74dda810d4b3f5201053a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c5587c5ecdc74dda810d4b3f5201053a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5587c5ecdc74dda810d4b3f5201053a" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 40255 } } }
I20260929 20:22:20.971943 17716 sys_catalog.cc:455] T 00000000000000000000000000000000 P c5587c5ecdc74dda810d4b3f5201053a [sys.catalog]: SysCatalogTable state changed. Reason: New leader c5587c5ecdc74dda810d4b3f5201053a. Latest consensus state: current_term: 1 leader_uuid: "c5587c5ecdc74dda810d4b3f5201053a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5587c5ecdc74dda810d4b3f5201053a" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 40255 } } }
I20260929 20:22:20.972455 17716 sys_catalog.cc:458] T 00000000000000000000000000000000 P c5587c5ecdc74dda810d4b3f5201053a [sys.catalog]: This master's current role is: LEADER
I20260929 20:22:20.973083 17715 sys_catalog.cc:458] T 00000000000000000000000000000000 P c5587c5ecdc74dda810d4b3f5201053a [sys.catalog]: This master's current role is: LEADER
I20260929 20:22:20.974267 17726 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:22:20.979215 17726 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:22:20.988099 17726 catalog_manager.cc:1347] Generated new cluster ID: e4b59d0004bd4c8b82431083db027ee2
I20260929 20:22:20.988270 17726 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:22:21.012758 17726 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 20:22:21.014182 17726 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:22:21.022544 17726 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c5587c5ecdc74dda810d4b3f5201053a: Generated new TSK 0
I20260929 20:22:21.023582 17726 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.17.53.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.53.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.53.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.65.keytab.
Entry for principal kudu/127.17.53.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.65.keytab.
I20260929 20:22:21.082458 17621 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
/tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790713339161492-17621-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790713339161492-17621-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790713339161492-17621-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790713339161492-17621-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.53.65:0
--local_ip_for_outbound_sockets=127.17.53.65
--webserver_interface=127.17.53.65
--webserver_port=0
--tserver_master_addrs=127.17.53.126:40255
--keytab_file=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.65.keytab
--principal=kudu/127.17.53.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.53.84:33983
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790713339161492-17621-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790713339161492-17621-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790713339161492-17621-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:22:21.411700 17735 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:22:21.411967 17735 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:22:21.412140 17735 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:22:21.419471 17735 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:22:21.419605 17735 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:22:21.419790 17735 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.53.65
I20260929 20:22:21.429785 17735 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.53.84:33983
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790713339161492-17621-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790713339161492-17621-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.65.keytab
--principal=kudu/127.17.53.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.53.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790713339161492-17621-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.17.53.65
--webserver_port=0
--tserver_master_addrs=127.17.53.126:40255
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.53.65
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790713339161492-17621-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:22:21.431149 17735 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:22:21.432468 17735 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:22:21.445763 17740 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:22:21.445708 17741 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:22:21.445681 17743 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:22:21.446779 17735 server_base.cc:1034] running on GCE node
I20260929 20:22:21.447394 17735 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:22:21.448304 17735 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:22:21.449496 17735 hybrid_clock.cc:648] HybridClock initialized: now 1790713341449435 us; error 51 us; skew 500 ppm
I20260929 20:22:21.449808 17735 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:22:21 dist-test-slave-zbnc krb5kdc[17628](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713341, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.53.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:22:21.453014 17735 init.cc:377] Logged in from keytab as kudu/127.17.53.65@KRBTEST.COM (short username kudu)
I20260929 20:22:21.454527 17735 webserver.cc:460] Webserver started at http://127.17.53.65:33239/ using document root <none> and password file <none>
I20260929 20:22:21.455279 17735 fs_manager.cc:359] Metadata directory not provided
I20260929 20:22:21.455406 17735 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:22:21.455718 17735 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:22:21.458554 17735 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790713339161492-17621-0/minicluster-data/ts-0/data/instance:
uuid: "b74c0082b9134c4aaacb246a87352771"
format_stamp: "Formatted at 2026-09-29 20:22:21 on dist-test-slave-zbnc"
server_key: "c557b80c2157b11fef2b96d5cdb0a558"
server_key_iv: "f46c92c3f14288e7b9d1bbe710f387e6"
server_key_version: "encryptionkey@0"
I20260929 20:22:21.459352 17735 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790713339161492-17621-0/minicluster-data/ts-0/wal/instance:
uuid: "b74c0082b9134c4aaacb246a87352771"
format_stamp: "Formatted at 2026-09-29 20:22:21 on dist-test-slave-zbnc"
server_key: "c557b80c2157b11fef2b96d5cdb0a558"
server_key_iv: "f46c92c3f14288e7b9d1bbe710f387e6"
server_key_version: "encryptionkey@0"
I20260929 20:22:21.464684 17735 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260929 20:22:21.468122 17750 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:22:21.469564 17735 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 20:22:21.469686 17735 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790713339161492-17621-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790713339161492-17621-0/minicluster-data/ts-0/wal
uuid: "b74c0082b9134c4aaacb246a87352771"
format_stamp: "Formatted at 2026-09-29 20:22:21 on dist-test-slave-zbnc"
server_key: "c557b80c2157b11fef2b96d5cdb0a558"
server_key_iv: "f46c92c3f14288e7b9d1bbe710f387e6"
server_key_version: "encryptionkey@0"
I20260929 20:22:21.469844 17735 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790713339161492-17621-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790713339161492-17621-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790713339161492-17621-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:22:21.482772 17735 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:22:21.486505 17735 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:22:21.486743 17735 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:22:21.487694 17735 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:22:21.488965 17735 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:22:21.489035 17735 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:22:21.489192 17735 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:22:21.489274 17735 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:22:21.522922 17735 rpc_server.cc:307] RPC server started. Bound to: 127.17.53.65:36027
I20260929 20:22:21.522944 17863 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.53.65:36027 every 8 connection(s)
I20260929 20:22:21.524211 17735 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790713339161492-17621-0/minicluster-data/ts-0/data/info.pb
I20260929 20:22:21.528808 17621 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu as pid 17735
I20260929 20:22:21.529037 17621 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790713339161492-17621-0/minicluster-data/ts-0/wal/instance
I20260929 20:22:21.529882 17621 external_mini_cluster.cc:1428] Setting key ef7d92260b7d9b35c501bcffe79a8f72
Sep 29 20:22:21 dist-test-slave-zbnc krb5kdc[17628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713341, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.53.65@KRBTEST.COM for kudu/127.17.53.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.17.53.66@KRBTEST.COM; defaulting to no policy
I20260929 20:22:21.545701 17866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:21.526823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.53.65:32923 (local address 127.17.53.126:40255)
0929 20:22:21.527313 (+ 490us) server_negotiation.cc:206] Beginning negotiation
0929 20:22:21.527318 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:22:21.528173 (+ 855us) server_negotiation.cc:407] Connection header received
0929 20:22:21.530581 (+ 2408us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:22:21.530585 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:22:21.530650 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:22:21.530806 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:22:21.532658 (+ 1852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:21.533400 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:21.535005 (+ 1605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:21.535429 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:21.535565 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:22:21.539771 (+ 4206us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:22:21.539776 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:22:21.539779 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:22:21.539812 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:22:21.542462 (+ 2650us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:22:21.543285 (+ 823us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:22:21.543289 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:22:21.543291 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:22:21.543408 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:22:21.543837 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:22:21.543841 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:22:21.543849 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:22:21.544108 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:22:21.544274 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0929 20:22:21.545104 (+ 830us) server_negotiation.cc:299] Negotiation successful
0929 20:22:21.545345 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":384,"thread_start_us":256,"threads_started":1}
I20260929 20:22:21.546685 17867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:21.527331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:40255 (local address 127.17.53.65:32923)
0929 20:22:21.528010 (+ 679us) negotiation.cc:107] Waiting for socket to connect
0929 20:22:21.528059 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0929 20:22:21.530276 (+ 2217us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:22:21.531101 (+ 825us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:22:21.531113 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:22:21.531860 (+ 747us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:22:21.532493 (+ 633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:22:21.532511 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:22:21.533553 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:22:21.533558 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:22:21.534816 (+ 1258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:22:21.534832 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:22:21.535654 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:22:21.535663 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:22:21.535925 (+ 262us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:22:21.536623 (+ 698us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:22:21.536644 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:22:21.539535 (+ 2891us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:22:21.542604 (+ 3069us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:22:21.542613 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:22:21.542632 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:22:21.543151 (+ 519us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:22:21.543520 (+ 369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:22:21.543526 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:22:21.543529 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:22:21.543718 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:22:21.544238 (+ 520us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:22:21.544246 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:22:21.544629 (+ 383us) client_negotiation.cc:769] Sending connection context
0929 20:22:21.544902 (+ 273us) client_negotiation.cc:240] Negotiation successful
0929 20:22:21.545454 (+ 552us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":461,"thread_start_us":264,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.53.66@KRBTEST.COM" created.
I20260929 20:22:21.548390 17864 heartbeater.cc:347] Connected to a master server at 127.17.53.126:40255
I20260929 20:22:21.548776 17864 heartbeater.cc:464] Registering TS with master...
I20260929 20:22:21.549737 17864 heartbeater.cc:511] Master 127.17.53.126:40255 requested a full tablet report, sending...
I20260929 20:22:21.552271 17674 ts_manager.cc:194] Registered new tserver with Master: b74c0082b9134c4aaacb246a87352771 (127.17.53.65:36027)
I20260929 20:22:21.553743 17674 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.53.65@KRBTEST.COM'} at 127.17.53.65:32923
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.53.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.66.keytab.
Entry for principal kudu/127.17.53.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.66.keytab.
I20260929 20:22:21.569717 17621 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
/tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790713339161492-17621-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790713339161492-17621-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790713339161492-17621-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790713339161492-17621-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.53.66:0
--local_ip_for_outbound_sockets=127.17.53.66
--webserver_interface=127.17.53.66
--webserver_port=0
--tserver_master_addrs=127.17.53.126:40255
--keytab_file=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.66.keytab
--principal=kudu/127.17.53.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.53.84:33983
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790713339161492-17621-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790713339161492-17621-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790713339161492-17621-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:22:21.892905 17870 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:22:21.893190 17870 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:22:21.893301 17870 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:22:21.900530 17870 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:22:21.900653 17870 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:22:21.900777 17870 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.53.66
I20260929 20:22:21.910391 17870 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.53.84:33983
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790713339161492-17621-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790713339161492-17621-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.66.keytab
--principal=kudu/127.17.53.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.53.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790713339161492-17621-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.17.53.66
--webserver_port=0
--tserver_master_addrs=127.17.53.126:40255
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.53.66
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790713339161492-17621-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:22:21.911669 17870 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:22:21.913044 17870 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:22:21.926049 17876 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:22:21.926542 17878 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:22:21.926635 17875 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:22:21.927246 17870 server_base.cc:1034] running on GCE node
I20260929 20:22:21.927855 17870 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:22:21.928910 17870 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:22:21.930128 17870 hybrid_clock.cc:648] HybridClock initialized: now 1790713341930089 us; error 38 us; skew 500 ppm
I20260929 20:22:21.930442 17870 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:22:21 dist-test-slave-zbnc krb5kdc[17628](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713341, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.53.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:22:21.933635 17870 init.cc:377] Logged in from keytab as kudu/127.17.53.66@KRBTEST.COM (short username kudu)
I20260929 20:22:21.935210 17870 webserver.cc:460] Webserver started at http://127.17.53.66:44205/ using document root <none> and password file <none>
I20260929 20:22:21.935900 17870 fs_manager.cc:359] Metadata directory not provided
I20260929 20:22:21.936028 17870 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:22:21.936335 17870 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:22:21.939258 17870 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790713339161492-17621-0/minicluster-data/ts-1/data/instance:
uuid: "1c092cb47d414dcc853dcd6011c571b2"
format_stamp: "Formatted at 2026-09-29 20:22:21 on dist-test-slave-zbnc"
server_key: "319d1ec64a508b8c28bbf38dc8312e60"
server_key_iv: "1cc874358e8dce9a3cf74751eee5de41"
server_key_version: "encryptionkey@0"
I20260929 20:22:21.940011 17870 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790713339161492-17621-0/minicluster-data/ts-1/wal/instance:
uuid: "1c092cb47d414dcc853dcd6011c571b2"
format_stamp: "Formatted at 2026-09-29 20:22:21 on dist-test-slave-zbnc"
server_key: "319d1ec64a508b8c28bbf38dc8312e60"
server_key_iv: "1cc874358e8dce9a3cf74751eee5de41"
server_key_version: "encryptionkey@0"
I20260929 20:22:21.945467 17870 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260929 20:22:21.949112 17885 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:22:21.950568 17870 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 20:22:21.950759 17870 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790713339161492-17621-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790713339161492-17621-0/minicluster-data/ts-1/wal
uuid: "1c092cb47d414dcc853dcd6011c571b2"
format_stamp: "Formatted at 2026-09-29 20:22:21 on dist-test-slave-zbnc"
server_key: "319d1ec64a508b8c28bbf38dc8312e60"
server_key_iv: "1cc874358e8dce9a3cf74751eee5de41"
server_key_version: "encryptionkey@0"
I20260929 20:22:21.951036 17870 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790713339161492-17621-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790713339161492-17621-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790713339161492-17621-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:22:21.965365 17870 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:22:21.969830 17870 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:22:21.970124 17870 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:22:21.971096 17870 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:22:21.972373 17870 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:22:21.972445 17870 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:22:21.972612 17870 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:22:21.972679 17870 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:22:22.006633 17870 rpc_server.cc:307] RPC server started. Bound to: 127.17.53.66:37319
I20260929 20:22:22.006666 17998 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.53.66:37319 every 8 connection(s)
I20260929 20:22:22.007992 17870 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790713339161492-17621-0/minicluster-data/ts-1/data/info.pb
I20260929 20:22:22.016705 17621 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu as pid 17870
I20260929 20:22:22.016901 17621 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790713339161492-17621-0/minicluster-data/ts-1/wal/instance
I20260929 20:22:22.017345 17621 external_mini_cluster.cc:1428] Setting key 1bb734ec607aa1a60291d9a7e21b044a
Sep 29 20:22:22 dist-test-slave-zbnc krb5kdc[17628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713341, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.53.66@KRBTEST.COM for kudu/127.17.53.126@KRBTEST.COM
I20260929 20:22:22.029592 17866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:22.010750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.53.66:51143 (local address 127.17.53.126:40255)
0929 20:22:22.010958 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0929 20:22:22.010963 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:22:22.012170 (+ 1207us) server_negotiation.cc:407] Connection header received
0929 20:22:22.013983 (+ 1813us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:22:22.013989 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:22:22.014080 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:22:22.014197 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:22:22.015722 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:22.016442 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:22.018225 (+ 1783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:22.018787 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:22.018955 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:22:22.022467 (+ 3512us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:22:22.022472 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:22:22.022478 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:22:22.022510 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:22:22.026060 (+ 3550us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:22:22.026885 (+ 825us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:22:22.026893 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:22:22.026899 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:22:22.027060 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:22:22.027511 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:22:22.027518 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:22:22.027524 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:22:22.027914 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:22:22.028060 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0929 20:22:22.028847 (+ 787us) server_negotiation.cc:299] Negotiation successful
0929 20:22:22.029111 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 20:22:22.030421 18001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:22.011229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:40255 (local address 127.17.53.66:51143)
0929 20:22:22.011984 (+ 755us) negotiation.cc:107] Waiting for socket to connect
0929 20:22:22.012022 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0929 20:22:22.013718 (+ 1696us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:22:22.014419 (+ 701us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:22:22.014427 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:22:22.014995 (+ 568us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:22:22.015562 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:22:22.015579 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:22:22.016602 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:22:22.016606 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:22:22.018056 (+ 1450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:22:22.018070 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:22:22.019052 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:22:22.019059 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:22:22.019394 (+ 335us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:22:22.020120 (+ 726us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:22:22.020143 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:22:22.022236 (+ 2093us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:22:22.026222 (+ 3986us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:22:22.026229 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:22:22.026242 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:22:22.026736 (+ 494us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:22:22.027198 (+ 462us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:22:22.027203 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:22:22.027206 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:22:22.027393 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:22:22.028079 (+ 686us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:22:22.028085 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:22:22.028433 (+ 348us) client_negotiation.cc:769] Sending connection context
0929 20:22:22.028709 (+ 276us) client_negotiation.cc:240] Negotiation successful
0929 20:22:22.029139 (+ 430us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":522,"thread_start_us":319,"threads_started":1}
I20260929 20:22:22.032172 17999 heartbeater.cc:347] Connected to a master server at 127.17.53.126:40255
I20260929 20:22:22.032519 17999 heartbeater.cc:464] Registering TS with master...
I20260929 20:22:22.033265 17999 heartbeater.cc:511] Master 127.17.53.126:40255 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.17.53.67@KRBTEST.COM; defaulting to no policy
I20260929 20:22:22.034819 17674 ts_manager.cc:194] Registered new tserver with Master: 1c092cb47d414dcc853dcd6011c571b2 (127.17.53.66:37319)
I20260929 20:22:22.035684 17674 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.53.66@KRBTEST.COM'} at 127.17.53.66:51143
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.53.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.53.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.67.keytab.
Entry for principal kudu/127.17.53.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.67.keytab.
I20260929 20:22:22.059069 17621 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
/tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790713339161492-17621-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790713339161492-17621-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790713339161492-17621-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790713339161492-17621-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.53.67:0
--local_ip_for_outbound_sockets=127.17.53.67
--webserver_interface=127.17.53.67
--webserver_port=0
--tserver_master_addrs=127.17.53.126:40255
--keytab_file=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.67.keytab
--principal=kudu/127.17.53.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.53.84:33983
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790713339161492-17621-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790713339161492-17621-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790713339161492-17621-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:22:22.381268 18004 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:22:22.381544 18004 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:22:22.381641 18004 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:22:22.389920 18004 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:22:22.390044 18004 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:22:22.390168 18004 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.53.67
I20260929 20:22:22.400199 18004 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.53.84:33983
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790713339161492-17621-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790713339161492-17621-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.67.keytab
--principal=kudu/127.17.53.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.53.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790713339161492-17621-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.17.53.67
--webserver_port=0
--tserver_master_addrs=127.17.53.126:40255
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.53.67
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790713339161492-17621-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:22:22.401485 18004 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:22:22.402794 18004 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:22:22.416193 18012 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:22:22.416288 18009 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:22:22.416275 18010 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:22:22.420552 18004 server_base.cc:1034] running on GCE node
I20260929 20:22:22.421173 18004 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:22:22.422125 18004 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:22:22.423341 18004 hybrid_clock.cc:648] HybridClock initialized: now 1790713342423275 us; error 40 us; skew 500 ppm
I20260929 20:22:22.423681 18004 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:22:22 dist-test-slave-zbnc krb5kdc[17628](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713342, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.53.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:22:22.426821 18004 init.cc:377] Logged in from keytab as kudu/127.17.53.67@KRBTEST.COM (short username kudu)
I20260929 20:22:22.428431 18004 webserver.cc:460] Webserver started at http://127.17.53.67:41735/ using document root <none> and password file <none>
I20260929 20:22:22.429174 18004 fs_manager.cc:359] Metadata directory not provided
I20260929 20:22:22.429317 18004 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:22:22.429625 18004 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:22:22.432657 18004 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790713339161492-17621-0/minicluster-data/ts-2/data/instance:
uuid: "6572fcc238b2461c93f16cd59efb6370"
format_stamp: "Formatted at 2026-09-29 20:22:22 on dist-test-slave-zbnc"
server_key: "f1c66b5abbbaf507c3fde813e54c43ee"
server_key_iv: "ae4e80dd09964f9c8392691465bfbd69"
server_key_version: "encryptionkey@0"
I20260929 20:22:22.433434 18004 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790713339161492-17621-0/minicluster-data/ts-2/wal/instance:
uuid: "6572fcc238b2461c93f16cd59efb6370"
format_stamp: "Formatted at 2026-09-29 20:22:22 on dist-test-slave-zbnc"
server_key: "f1c66b5abbbaf507c3fde813e54c43ee"
server_key_iv: "ae4e80dd09964f9c8392691465bfbd69"
server_key_version: "encryptionkey@0"
I20260929 20:22:22.438548 18004 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 20:22:22.442026 18019 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:22:22.443456 18004 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 20:22:22.443632 18004 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790713339161492-17621-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790713339161492-17621-0/minicluster-data/ts-2/wal
uuid: "6572fcc238b2461c93f16cd59efb6370"
format_stamp: "Formatted at 2026-09-29 20:22:22 on dist-test-slave-zbnc"
server_key: "f1c66b5abbbaf507c3fde813e54c43ee"
server_key_iv: "ae4e80dd09964f9c8392691465bfbd69"
server_key_version: "encryptionkey@0"
I20260929 20:22:22.443841 18004 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790713339161492-17621-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790713339161492-17621-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790713339161492-17621-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:22:22.465071 18004 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:22:22.469445 18004 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:22:22.469743 18004 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:22:22.470832 18004 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:22:22.472227 18004 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:22:22.472338 18004 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:22:22.472479 18004 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:22:22.472551 18004 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:22:22.506644 18004 rpc_server.cc:307] RPC server started. Bound to: 127.17.53.67:37659
I20260929 20:22:22.506726 18131 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.53.67:37659 every 8 connection(s)
I20260929 20:22:22.507974 18004 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790713339161492-17621-0/minicluster-data/ts-2/data/info.pb
I20260929 20:22:22.515635 17621 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu as pid 18004
I20260929 20:22:22.515828 17621 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790713339161492-17621-0/minicluster-data/ts-2/wal/instance
I20260929 20:22:22.516266 17621 external_mini_cluster.cc:1428] Setting key dbec41709190df2de9d7c239cf6669c4
Sep 29 20:22:22 dist-test-slave-zbnc krb5kdc[17628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713342, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.53.67@KRBTEST.COM for kudu/127.17.53.126@KRBTEST.COM
I20260929 20:22:22.528841 17866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:22.510548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.53.67:46795 (local address 127.17.53.126:40255)
0929 20:22:22.510728 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0929 20:22:22.510733 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:22:22.511871 (+ 1138us) server_negotiation.cc:407] Connection header received
0929 20:22:22.513793 (+ 1922us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:22:22.513802 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:22:22.513889 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:22:22.514000 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:22:22.515553 (+ 1553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:22.516486 (+ 933us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:22.518287 (+ 1801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:22.518915 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:22.519049 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:22:22.522532 (+ 3483us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:22:22.522539 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:22:22.522544 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:22:22.522583 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:22:22.525854 (+ 3271us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:22:22.526646 (+ 792us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:22:22.526652 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:22:22.526656 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:22:22.526812 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:22:22.527224 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:22:22.527228 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:22:22.527232 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:22:22.527579 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:22:22.527732 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0929 20:22:22.528347 (+ 615us) server_negotiation.cc:299] Negotiation successful
0929 20:22:22.528565 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 20:22:22.529709 18134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:22.511016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:40255 (local address 127.17.53.67:46795)
0929 20:22:22.511694 (+ 678us) negotiation.cc:107] Waiting for socket to connect
0929 20:22:22.511742 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0929 20:22:22.513468 (+ 1726us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:22:22.514247 (+ 779us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:22:22.514256 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:22:22.514767 (+ 511us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:22:22.515374 (+ 607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:22:22.515392 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:22:22.516628 (+ 1236us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:22:22.516634 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:22:22.518147 (+ 1513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:22:22.518159 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:22:22.519113 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:22:22.519118 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:22:22.519370 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:22:22.520103 (+ 733us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:22:22.520130 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:22:22.522328 (+ 2198us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:22:22.525991 (+ 3663us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:22:22.525998 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:22:22.526010 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:22:22.526532 (+ 522us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:22:22.526952 (+ 420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:22:22.526955 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:22:22.526958 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:22:22.527129 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:22:22.527699 (+ 570us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:22:22.527705 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:22:22.528007 (+ 302us) client_negotiation.cc:769] Sending connection context
0929 20:22:22.528225 (+ 218us) client_negotiation.cc:240] Negotiation successful
0929 20:22:22.528615 (+ 390us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":452,"thread_start_us":258,"threads_started":1}
I20260929 20:22:22.531180 18132 heartbeater.cc:347] Connected to a master server at 127.17.53.126:40255
I20260929 20:22:22.531491 18132 heartbeater.cc:464] Registering TS with master...
I20260929 20:22:22.532187 18132 heartbeater.cc:511] Master 127.17.53.126:40255 requested a full tablet report, sending...
I20260929 20:22:22.533546 17674 ts_manager.cc:194] Registered new tserver with Master: 6572fcc238b2461c93f16cd59efb6370 (127.17.53.67:37659)
I20260929 20:22:22.534338 17674 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.53.67@KRBTEST.COM'} at 127.17.53.67:46795
I20260929 20:22:22.545075 17621 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 20:22:22.556699 17864 heartbeater.cc:503] Master 127.17.53.126:40255 was elected leader, sending a full tablet report...
Sep 29 20:22:22 dist-test-slave-zbnc krb5kdc[17628](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713342, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 29 20:22:22 dist-test-slave-zbnc krb5kdc[17628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713342, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.17.53.65@KRBTEST.COM
I20260929 20:22:22.591675 18139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:22.575528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51446 (local address 127.17.53.65:36027)
0929 20:22:22.576408 (+ 880us) server_negotiation.cc:206] Beginning negotiation
0929 20:22:22.576416 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 20:22:22.576446 (+ 30us) server_negotiation.cc:407] Connection header received
0929 20:22:22.576510 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:22:22.576518 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:22:22.576828 (+ 310us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:22:22.576969 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:22:22.578320 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:22.579667 (+ 1347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:22.580875 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:22.581501 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:22.581615 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:22:22.585567 (+ 3952us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:22:22.585580 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:22:22.585596 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:22:22.585636 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:22:22.588821 (+ 3185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:22:22.589440 (+ 619us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:22:22.589448 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:22:22.589450 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:22:22.589655 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:22:22.590040 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:22:22.590047 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:22:22.590049 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:22:22.590420 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:22:22.590677 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0929 20:22:22.590967 (+ 290us) server_negotiation.cc:299] Negotiation successful
0929 20:22:22.591203 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":709,"thread_start_us":573,"threads_started":1}
W20260929 20:22:22.593060 17778 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51446
Sep 29 20:22:22 dist-test-slave-zbnc krb5kdc[17628](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713342, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 29 20:22:22 dist-test-slave-zbnc krb5kdc[17628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713342, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.53.65@KRBTEST.COM
I20260929 20:22:22.638123 18139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:22.624417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51452 (local address 127.17.53.65:36027)
0929 20:22:22.624659 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0929 20:22:22.624666 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:22:22.624985 (+ 319us) server_negotiation.cc:407] Connection header received
0929 20:22:22.625200 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:22:22.625205 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:22:22.625269 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:22:22.625370 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:22:22.626663 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:22.627394 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:22.628625 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:22.629049 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:22.629189 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:22:22.633144 (+ 3955us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:22:22.633149 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:22:22.633152 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:22:22.633191 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:22:22.635696 (+ 2505us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:22:22.636305 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:22:22.636316 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:22:22.636318 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:22:22.636435 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:22:22.636796 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:22:22.636802 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:22:22.636806 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:22:22.637047 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:22:22.637293 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0929 20:22:22.637584 (+ 291us) server_negotiation.cc:299] Negotiation successful
0929 20:22:22.637810 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260929 20:22:22.639622 17621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu pid 17735
I20260929 20:22:22.651834 18139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:22.640547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51462 (local address 127.17.53.65:36027)
0929 20:22:22.640719 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0929 20:22:22.640727 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 20:22:22.641054 (+ 327us) server_negotiation.cc:407] Connection header received
0929 20:22:22.641233 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:22:22.641238 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:22:22.641330 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:22:22.641438 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:22:22.642533 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:22.643292 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:22.644461 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:22.644896 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:22.644987 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:22:22.647259 (+ 2272us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:22:22.647263 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:22:22.647266 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:22:22.647293 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:22:22.649743 (+ 2450us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:22:22.650303 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:22:22.650306 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:22:22.650308 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:22:22.650421 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:22:22.650704 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:22:22.650707 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:22:22.650709 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:22:22.650998 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:22:22.651170 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0929 20:22:22.651415 (+ 245us) server_negotiation.cc:299] Negotiation successful
0929 20:22:22.651558 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260929 20:22:22.652565 17859 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:22:22.739751 17621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu with pid 17735
I20260929 20:22:22.769969 17621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu pid 17870
Sep 29 20:22:22 dist-test-slave-zbnc krb5kdc[17628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713342, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.53.66@KRBTEST.COM
I20260929 20:22:22.787317 18146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:22.771095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56320 (local address 127.17.53.66:37319)
0929 20:22:22.771876 (+ 781us) server_negotiation.cc:206] Beginning negotiation
0929 20:22:22.771883 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:22:22.771910 (+ 27us) server_negotiation.cc:407] Connection header received
0929 20:22:22.771992 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:22:22.771998 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:22:22.772238 (+ 240us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:22:22.772419 (+ 181us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:22:22.773642 (+ 1223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:22.774746 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:22.776140 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:22.776716 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:22.776851 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:22:22.781417 (+ 4566us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:22:22.781428 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:22:22.781442 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:22:22.781475 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:22:22.784463 (+ 2988us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:22:22.785112 (+ 649us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:22:22.785119 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:22:22.785121 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:22:22.785245 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:22:22.785672 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:22:22.785681 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:22:22.785683 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:22:22.786017 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:22:22.786345 (+ 328us) server_negotiation.cc:1035] Waiting for connection context
0929 20:22:22.786627 (+ 282us) server_negotiation.cc:299] Negotiation successful
0929 20:22:22.786916 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":619,"thread_start_us":495,"threads_started":1}
I20260929 20:22:22.788599 17994 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:22:22.872123 17621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu with pid 17870
I20260929 20:22:22.891927 17621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu pid 18004
Sep 29 20:22:22 dist-test-slave-zbnc krb5kdc[17628](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713342, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.53.67@KRBTEST.COM
I20260929 20:22:22.907444 18148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:22.892892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60894 (local address 127.17.53.67:37659)
0929 20:22:22.893579 (+ 687us) server_negotiation.cc:206] Beginning negotiation
0929 20:22:22.893586 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:22:22.893615 (+ 29us) server_negotiation.cc:407] Connection header received
0929 20:22:22.893674 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:22:22.893681 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:22:22.893905 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:22:22.894029 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:22:22.895324 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:22.896354 (+ 1030us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:22.897540 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:22.898127 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:22.898222 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:22:22.902238 (+ 4016us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:22:22.902250 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:22:22.902267 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:22:22.902305 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:22:22.905088 (+ 2783us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:22:22.905679 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:22:22.905687 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:22:22.905689 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:22:22.905814 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:22:22.906100 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:22:22.906103 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:22:22.906105 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:22:22.906411 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:22:22.906623 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0929 20:22:22.906845 (+ 222us) server_negotiation.cc:299] Negotiation successful
0929 20:22:22.907099 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":542,"thread_start_us":381,"threads_started":1}
I20260929 20:22:22.908690 18127 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:22:22.990767 17621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu with pid 18004
I20260929 20:22:23.007468 17621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu pid 17641
I20260929 20:22:23.008903 17704 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:22:23.115679 17621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu with pid 17641
2026-09-29T20:22:23Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (3920 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790713339161492-17621-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 20:22:23 dist-test-slave-zbnc krb5kdc[18153](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 20:22:23 dist-test-slave-zbnc krb5kdc[18153](info): set up 2 sockets
Sep 29 20:22:23 dist-test-slave-zbnc krb5kdc[18153](info): commencing operation
krb5kdc: starting...
W20260929 20:22:24.212497 17621 mini_kdc.cc:121] Time spent starting KDC: real 1.055s user 0.001s sys 0.013s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790713339161492-17621-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790713339161492-17621-0/krb5kdc/test-user.keytab.
Sep 29 20:22:24 dist-test-slave-zbnc krb5kdc[18153](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713344, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T20:22:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T20:22:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.17.53.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.53.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.53.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.126.keytab.
Entry for principal kudu/127.17.53.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.126.keytab.
I20260929 20:22:24.391875 17621 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
/tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790713339161492-17621-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790713339161492-17621-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790713339161492-17621-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790713339161492-17621-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.53.126:44579
--webserver_interface=127.17.53.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.126.keytab
--principal=kudu/127.17.53.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.53.84:35919
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.53.126:44579
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790713339161492-17621-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790713339161492-17621-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790713339161492-17621-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:22:24.713887 18167 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:22:24.714196 18167 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:22:24.714380 18167 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:22:24.721774 18167 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:22:24.721896 18167 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:22:24.721997 18167 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:22:24.722095 18167 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:22:24.722169 18167 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:22:24.732201 18167 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.53.84:35919
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790713339161492-17621-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790713339161492-17621-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.53.126:44579
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.126.keytab
--principal=kudu/127.17.53.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.53.126:44579
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790713339161492-17621-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.53.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790713339161492-17621-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:22:24.733579 18167 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:22:24.734959 18167 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:22:24.746666 18175 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:22:24.746632 18172 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:22:24.746703 18173 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:22:24.747605 18167 server_base.cc:1034] running on GCE node
I20260929 20:22:24.748347 18167 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:22:24.749783 18167 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:22:24.750981 18167 hybrid_clock.cc:648] HybridClock initialized: now 1790713344750920 us; error 40 us; skew 500 ppm
I20260929 20:22:24.751358 18167 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:22:24 dist-test-slave-zbnc krb5kdc[18153](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713344, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.53.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:22:24.759652 18167 init.cc:377] Logged in from keytab as kudu/127.17.53.126@KRBTEST.COM (short username kudu)
I20260929 20:22:24.761286 18167 webserver.cc:460] Webserver started at http://127.17.53.126:42001/ using document root <none> and password file <none>
I20260929 20:22:24.761996 18167 fs_manager.cc:359] Metadata directory not provided
I20260929 20:22:24.762141 18167 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:22:24.762468 18167 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:22:24.765501 18167 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790713339161492-17621-0/minicluster-data/master-0/data/instance:
uuid: "3e2da1b2cbd24cf8aaec69a52709c2bb"
format_stamp: "Formatted at 2026-09-29 20:22:24 on dist-test-slave-zbnc"
server_key: "f137fc0c05b077a24d75cfa3b0ba41ca"
server_key_iv: "bee78e8e2b7cd4c9567724f2e37c9146"
server_key_version: "encryptionkey@0"
I20260929 20:22:24.766248 18167 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790713339161492-17621-0/minicluster-data/master-0/wal/instance:
uuid: "3e2da1b2cbd24cf8aaec69a52709c2bb"
format_stamp: "Formatted at 2026-09-29 20:22:24 on dist-test-slave-zbnc"
server_key: "f137fc0c05b077a24d75cfa3b0ba41ca"
server_key_iv: "bee78e8e2b7cd4c9567724f2e37c9146"
server_key_version: "encryptionkey@0"
I20260929 20:22:24.771378 18167 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260929 20:22:24.774664 18183 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:22:24.776005 18167 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 20:22:24.776177 18167 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790713339161492-17621-0/minicluster-data/master-0/data,/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790713339161492-17621-0/minicluster-data/master-0/wal
uuid: "3e2da1b2cbd24cf8aaec69a52709c2bb"
format_stamp: "Formatted at 2026-09-29 20:22:24 on dist-test-slave-zbnc"
server_key: "f137fc0c05b077a24d75cfa3b0ba41ca"
server_key_iv: "bee78e8e2b7cd4c9567724f2e37c9146"
server_key_version: "encryptionkey@0"
I20260929 20:22:24.776372 18167 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790713339161492-17621-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790713339161492-17621-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790713339161492-17621-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:22:24.803213 18167 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:22:24.807168 18167 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:22:24.807478 18167 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:22:24.825578 18167 rpc_server.cc:307] RPC server started. Bound to: 127.17.53.126:44579
I20260929 20:22:24.825601 18234 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.53.126:44579 every 8 connection(s)
I20260929 20:22:24.827028 18167 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790713339161492-17621-0/minicluster-data/master-0/data/info.pb
I20260929 20:22:24.828401 17621 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu as pid 18167
I20260929 20:22:24.828639 17621 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790713339161492-17621-0/minicluster-data/master-0/wal/instance
I20260929 20:22:24.829476 17621 external_mini_cluster.cc:1428] Setting key db1dd6262f9a5d88675fe5899a906be0
I20260929 20:22:24.832032 18235 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 29 20:22:24 dist-test-slave-zbnc krb5kdc[18153](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713344, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.53.126@KRBTEST.COM
I20260929 20:22:24.845223 18235 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3e2da1b2cbd24cf8aaec69a52709c2bb: Bootstrap starting.
I20260929 20:22:24.849416 18235 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3e2da1b2cbd24cf8aaec69a52709c2bb: Neither blocks nor log segments found. Creating new log.
I20260929 20:22:24.850735 18235 log.cc:824] T 00000000000000000000000000000000 P 3e2da1b2cbd24cf8aaec69a52709c2bb: Log is configured to *not* fsync() on all Append() calls
I20260929 20:22:24.853626 18238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:24.831618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50940 (local address 127.17.53.126:44579)
0929 20:22:24.832429 (+ 811us) server_negotiation.cc:206] Beginning negotiation
0929 20:22:24.832442 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 20:22:24.832482 (+ 40us) server_negotiation.cc:407] Connection header received
0929 20:22:24.834162 (+ 1680us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:22:24.834183 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:22:24.834585 (+ 402us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:22:24.834982 (+ 397us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:22:24.836809 (+ 1827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:24.838173 (+ 1364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:24.839881 (+ 1708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:24.840521 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:24.840679 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:22:24.845191 (+ 4512us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:22:24.845217 (+ 26us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:22:24.845235 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:22:24.845282 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:22:24.849071 (+ 3789us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:22:24.849804 (+ 733us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:22:24.849815 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:22:24.849826 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:22:24.850040 (+ 214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:22:24.850383 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:22:24.850393 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:22:24.850398 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:22:24.851107 (+ 709us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:22:24.851446 (+ 339us) server_negotiation.cc:1035] Waiting for connection context
0929 20:22:24.851918 (+ 472us) server_negotiation.cc:299] Negotiation successful
0929 20:22:24.852348 (+ 430us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":600,"thread_start_us":377,"threads_started":1}
I20260929 20:22:24.854156 18235 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3e2da1b2cbd24cf8aaec69a52709c2bb: No bootstrap required, opened a new log
I20260929 20:22:24.861604 18235 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3e2da1b2cbd24cf8aaec69a52709c2bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e2da1b2cbd24cf8aaec69a52709c2bb" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 44579 } }
I20260929 20:22:24.861992 18235 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3e2da1b2cbd24cf8aaec69a52709c2bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:22:24.862112 18235 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3e2da1b2cbd24cf8aaec69a52709c2bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e2da1b2cbd24cf8aaec69a52709c2bb, State: Initialized, Role: FOLLOWER
I20260929 20:22:24.862736 18235 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3e2da1b2cbd24cf8aaec69a52709c2bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e2da1b2cbd24cf8aaec69a52709c2bb" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 44579 } }
I20260929 20:22:24.863040 18235 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3e2da1b2cbd24cf8aaec69a52709c2bb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 20:22:24.863185 18235 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3e2da1b2cbd24cf8aaec69a52709c2bb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 20:22:24.863363 18235 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3e2da1b2cbd24cf8aaec69a52709c2bb [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:22:24.865465 18235 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3e2da1b2cbd24cf8aaec69a52709c2bb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e2da1b2cbd24cf8aaec69a52709c2bb" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 44579 } }
I20260929 20:22:24.865959 18235 leader_election.cc:302] T 00000000000000000000000000000000 P 3e2da1b2cbd24cf8aaec69a52709c2bb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3e2da1b2cbd24cf8aaec69a52709c2bb; no voters:
I20260929 20:22:24.866619 18235 leader_election.cc:288] T 00000000000000000000000000000000 P 3e2da1b2cbd24cf8aaec69a52709c2bb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 20:22:24.866765 18240 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3e2da1b2cbd24cf8aaec69a52709c2bb [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:22:24.867635 18240 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3e2da1b2cbd24cf8aaec69a52709c2bb [term 1 LEADER]: Becoming Leader. State: Replica: 3e2da1b2cbd24cf8aaec69a52709c2bb, State: Running, Role: LEADER
I20260929 20:22:24.868177 18240 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3e2da1b2cbd24cf8aaec69a52709c2bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e2da1b2cbd24cf8aaec69a52709c2bb" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 44579 } }
I20260929 20:22:24.868777 18235 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3e2da1b2cbd24cf8aaec69a52709c2bb [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:22:24.871109 18241 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e2da1b2cbd24cf8aaec69a52709c2bb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3e2da1b2cbd24cf8aaec69a52709c2bb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e2da1b2cbd24cf8aaec69a52709c2bb" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 44579 } } }
I20260929 20:22:24.871476 18241 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e2da1b2cbd24cf8aaec69a52709c2bb [sys.catalog]: This master's current role is: LEADER
I20260929 20:22:24.872337 18247 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:22:24.872121 18242 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e2da1b2cbd24cf8aaec69a52709c2bb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3e2da1b2cbd24cf8aaec69a52709c2bb. Latest consensus state: current_term: 1 leader_uuid: "3e2da1b2cbd24cf8aaec69a52709c2bb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e2da1b2cbd24cf8aaec69a52709c2bb" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 44579 } } }
I20260929 20:22:24.872546 18242 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e2da1b2cbd24cf8aaec69a52709c2bb [sys.catalog]: This master's current role is: LEADER
I20260929 20:22:24.877995 18247 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:22:24.890516 18247 catalog_manager.cc:1347] Generated new cluster ID: 8820b40b57d7421ea7a9b1c293cd75b1
I20260929 20:22:24.890642 18247 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:22:24.897797 18247 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 20:22:24.899398 18247 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:22:24.905282 18247 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3e2da1b2cbd24cf8aaec69a52709c2bb: Generated new TSK 0
I20260929 20:22:24.906251 18247 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.17.53.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.53.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.53.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.65.keytab.
Entry for principal kudu/127.17.53.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.65.keytab.
I20260929 20:22:24.952383 17621 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
/tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790713339161492-17621-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790713339161492-17621-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790713339161492-17621-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790713339161492-17621-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.53.65:0
--local_ip_for_outbound_sockets=127.17.53.65
--webserver_interface=127.17.53.65
--webserver_port=0
--tserver_master_addrs=127.17.53.126:44579
--keytab_file=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.65.keytab
--principal=kudu/127.17.53.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.53.84:35919
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790713339161492-17621-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790713339161492-17621-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790713339161492-17621-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:22:25.285054 18262 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:22:25.285372 18262 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:22:25.285528 18262 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:22:25.292659 18262 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:22:25.292805 18262 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:22:25.292965 18262 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.53.65
I20260929 20:22:25.303198 18262 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.53.84:35919
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790713339161492-17621-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790713339161492-17621-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.65.keytab
--principal=kudu/127.17.53.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.53.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790713339161492-17621-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.17.53.65
--webserver_port=0
--tserver_master_addrs=127.17.53.126:44579
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.53.65
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790713339161492-17621-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:22:25.304559 18262 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:22:25.305934 18262 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:22:25.319077 18270 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:22:25.319049 18268 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:22:25.319130 18267 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:22:25.319799 18262 server_base.cc:1034] running on GCE node
I20260929 20:22:25.320407 18262 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:22:25.321362 18262 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:22:25.322655 18262 hybrid_clock.cc:648] HybridClock initialized: now 1790713345322610 us; error 51 us; skew 500 ppm
I20260929 20:22:25.323030 18262 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:22:25 dist-test-slave-zbnc krb5kdc[18153](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713345, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.53.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:22:25.326160 18262 init.cc:377] Logged in from keytab as kudu/127.17.53.65@KRBTEST.COM (short username kudu)
I20260929 20:22:25.327684 18262 webserver.cc:460] Webserver started at http://127.17.53.65:39285/ using document root <none> and password file <none>
I20260929 20:22:25.328382 18262 fs_manager.cc:359] Metadata directory not provided
I20260929 20:22:25.328521 18262 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:22:25.328835 18262 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:22:25.331882 18262 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790713339161492-17621-0/minicluster-data/ts-0/data/instance:
uuid: "0da9582139ef4cfe91dbcbbe41311127"
format_stamp: "Formatted at 2026-09-29 20:22:25 on dist-test-slave-zbnc"
server_key: "61fe5cc6d557fc9990b2ea8e7f1c47fa"
server_key_iv: "3651ae56ed597866d8e2c977e0a6e439"
server_key_version: "encryptionkey@0"
I20260929 20:22:25.332670 18262 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790713339161492-17621-0/minicluster-data/ts-0/wal/instance:
uuid: "0da9582139ef4cfe91dbcbbe41311127"
format_stamp: "Formatted at 2026-09-29 20:22:25 on dist-test-slave-zbnc"
server_key: "61fe5cc6d557fc9990b2ea8e7f1c47fa"
server_key_iv: "3651ae56ed597866d8e2c977e0a6e439"
server_key_version: "encryptionkey@0"
I20260929 20:22:25.338001 18262 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 20:22:25.341562 18277 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:22:25.343058 18262 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 20:22:25.343230 18262 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790713339161492-17621-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790713339161492-17621-0/minicluster-data/ts-0/wal
uuid: "0da9582139ef4cfe91dbcbbe41311127"
format_stamp: "Formatted at 2026-09-29 20:22:25 on dist-test-slave-zbnc"
server_key: "61fe5cc6d557fc9990b2ea8e7f1c47fa"
server_key_iv: "3651ae56ed597866d8e2c977e0a6e439"
server_key_version: "encryptionkey@0"
I20260929 20:22:25.343442 18262 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790713339161492-17621-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790713339161492-17621-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790713339161492-17621-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:22:25.366768 18262 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:22:25.371258 18262 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:22:25.371569 18262 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:22:25.372546 18262 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:22:25.373929 18262 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:22:25.374037 18262 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:22:25.374185 18262 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:22:25.374257 18262 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:22:25.408521 18262 rpc_server.cc:307] RPC server started. Bound to: 127.17.53.65:45613
I20260929 20:22:25.408607 18390 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.53.65:45613 every 8 connection(s)
I20260929 20:22:25.409808 18262 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790713339161492-17621-0/minicluster-data/ts-0/data/info.pb
I20260929 20:22:25.411180 17621 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu as pid 18262
I20260929 20:22:25.411394 17621 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790713339161492-17621-0/minicluster-data/ts-0/wal/instance
I20260929 20:22:25.412005 17621 external_mini_cluster.cc:1428] Setting key 4bd476ecff7dd6b3ba98c0a455366dd0
Sep 29 20:22:25 dist-test-slave-zbnc krb5kdc[18153](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713345, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.53.65@KRBTEST.COM for kudu/127.17.53.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.17.53.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.53.66@KRBTEST.COM" created.
I20260929 20:22:25.433857 18393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:25.412380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.53.65:53305 (local address 127.17.53.126:44579)
0929 20:22:25.415187 (+ 2807us) server_negotiation.cc:206] Beginning negotiation
0929 20:22:25.415191 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:22:25.415208 (+ 17us) server_negotiation.cc:407] Connection header received
0929 20:22:25.417454 (+ 2246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:22:25.417458 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:22:25.417517 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:22:25.417610 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:22:25.419679 (+ 2069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:25.420891 (+ 1212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:25.422444 (+ 1553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:25.423236 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:25.423391 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:22:25.427182 (+ 3791us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:22:25.427193 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:22:25.427201 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:22:25.427252 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:22:25.430620 (+ 3368us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:22:25.431458 (+ 838us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:22:25.431462 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:22:25.431464 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:22:25.431579 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:22:25.431948 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:22:25.431951 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:22:25.431953 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:22:25.432284 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:22:25.432392 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0929 20:22:25.433234 (+ 842us) server_negotiation.cc:299] Negotiation successful
0929 20:22:25.433468 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2711,"thread_start_us":2611,"threads_started":1}
I20260929 20:22:25.434913 18394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:25.413174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:44579 (local address 127.17.53.65:53305)
0929 20:22:25.414312 (+ 1138us) negotiation.cc:107] Waiting for socket to connect
0929 20:22:25.414367 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0929 20:22:25.417152 (+ 2785us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:22:25.417895 (+ 743us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:22:25.417909 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:22:25.418746 (+ 837us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:22:25.419502 (+ 756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:22:25.419520 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:22:25.421047 (+ 1527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:22:25.421051 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:22:25.422298 (+ 1247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:22:25.422310 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:22:25.423450 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:22:25.423454 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:22:25.423650 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:22:25.424353 (+ 703us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:22:25.424376 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:22:25.426934 (+ 2558us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:22:25.430762 (+ 3828us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:22:25.430769 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:22:25.430783 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:22:25.431348 (+ 565us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:22:25.431669 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:22:25.431672 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:22:25.431674 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:22:25.431854 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:22:25.432388 (+ 534us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:22:25.432394 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:22:25.432841 (+ 447us) client_negotiation.cc:769] Sending connection context
0929 20:22:25.433111 (+ 270us) client_negotiation.cc:240] Negotiation successful
0929 20:22:25.433670 (+ 559us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":799,"thread_start_us":510,"threads_started":1}
I20260929 20:22:25.436553 18391 heartbeater.cc:347] Connected to a master server at 127.17.53.126:44579
I20260929 20:22:25.436857 18391 heartbeater.cc:464] Registering TS with master...
I20260929 20:22:25.437548 18391 heartbeater.cc:511] Master 127.17.53.126:44579 requested a full tablet report, sending...
I20260929 20:22:25.439625 18200 ts_manager.cc:194] Registered new tserver with Master: 0da9582139ef4cfe91dbcbbe41311127 (127.17.53.65:45613)
I20260929 20:22:25.441052 18200 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.53.65@KRBTEST.COM'} at 127.17.53.65:53305
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.53.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.66.keytab.
Entry for principal kudu/127.17.53.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.66.keytab.
I20260929 20:22:25.455044 17621 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
/tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790713339161492-17621-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790713339161492-17621-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790713339161492-17621-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790713339161492-17621-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.53.66:0
--local_ip_for_outbound_sockets=127.17.53.66
--webserver_interface=127.17.53.66
--webserver_port=0
--tserver_master_addrs=127.17.53.126:44579
--keytab_file=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.66.keytab
--principal=kudu/127.17.53.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.53.84:35919
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790713339161492-17621-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790713339161492-17621-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790713339161492-17621-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:22:25.777597 18397 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:22:25.777887 18397 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:22:25.777983 18397 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:22:25.785413 18397 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:22:25.785530 18397 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:22:25.785655 18397 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.53.66
I20260929 20:22:25.795444 18397 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.53.84:35919
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790713339161492-17621-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790713339161492-17621-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.66.keytab
--principal=kudu/127.17.53.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.53.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790713339161492-17621-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.17.53.66
--webserver_port=0
--tserver_master_addrs=127.17.53.126:44579
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.53.66
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790713339161492-17621-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:22:25.796657 18397 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:22:25.797909 18397 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:22:25.811079 18402 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:22:25.811111 18403 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:22:25.811781 18405 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:22:25.812017 18397 server_base.cc:1034] running on GCE node
I20260929 20:22:25.812619 18397 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:22:25.813582 18397 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:22:25.814808 18397 hybrid_clock.cc:648] HybridClock initialized: now 1790713345814738 us; error 78 us; skew 500 ppm
I20260929 20:22:25.815147 18397 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:22:25 dist-test-slave-zbnc krb5kdc[18153](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713345, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.53.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:22:25.818279 18397 init.cc:377] Logged in from keytab as kudu/127.17.53.66@KRBTEST.COM (short username kudu)
I20260929 20:22:25.819795 18397 webserver.cc:460] Webserver started at http://127.17.53.66:34083/ using document root <none> and password file <none>
I20260929 20:22:25.820497 18397 fs_manager.cc:359] Metadata directory not provided
I20260929 20:22:25.820636 18397 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:22:25.820950 18397 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:22:25.823948 18397 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790713339161492-17621-0/minicluster-data/ts-1/data/instance:
uuid: "ded15bfaed254bae892a2b39edf2915f"
format_stamp: "Formatted at 2026-09-29 20:22:25 on dist-test-slave-zbnc"
server_key: "bc1fe18cee416deff16ec225c8d735aa"
server_key_iv: "6d3d5db85fccbdd4464565077e8a2d1a"
server_key_version: "encryptionkey@0"
I20260929 20:22:25.824764 18397 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790713339161492-17621-0/minicluster-data/ts-1/wal/instance:
uuid: "ded15bfaed254bae892a2b39edf2915f"
format_stamp: "Formatted at 2026-09-29 20:22:25 on dist-test-slave-zbnc"
server_key: "bc1fe18cee416deff16ec225c8d735aa"
server_key_iv: "6d3d5db85fccbdd4464565077e8a2d1a"
server_key_version: "encryptionkey@0"
I20260929 20:22:25.830008 18397 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 20:22:25.833679 18412 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:22:25.835135 18397 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260929 20:22:25.835333 18397 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790713339161492-17621-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790713339161492-17621-0/minicluster-data/ts-1/wal
uuid: "ded15bfaed254bae892a2b39edf2915f"
format_stamp: "Formatted at 2026-09-29 20:22:25 on dist-test-slave-zbnc"
server_key: "bc1fe18cee416deff16ec225c8d735aa"
server_key_iv: "6d3d5db85fccbdd4464565077e8a2d1a"
server_key_version: "encryptionkey@0"
I20260929 20:22:25.835541 18397 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790713339161492-17621-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790713339161492-17621-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790713339161492-17621-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:22:25.847652 18397 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:22:25.851603 18397 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:22:25.851893 18397 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:22:25.852809 18397 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:22:25.854135 18397 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:22:25.854238 18397 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:22:25.854374 18397 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:22:25.854444 18397 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:22:25.888273 18397 rpc_server.cc:307] RPC server started. Bound to: 127.17.53.66:38565
I20260929 20:22:25.888334 18525 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.53.66:38565 every 8 connection(s)
I20260929 20:22:25.889595 18397 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790713339161492-17621-0/minicluster-data/ts-1/data/info.pb
I20260929 20:22:25.892623 17621 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu as pid 18397
I20260929 20:22:25.892838 17621 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790713339161492-17621-0/minicluster-data/ts-1/wal/instance
I20260929 20:22:25.893457 17621 external_mini_cluster.cc:1428] Setting key 9635cba6c46b47c5db44e80fe2fd1f80
Sep 29 20:22:25 dist-test-slave-zbnc krb5kdc[18153](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713345, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.53.66@KRBTEST.COM for kudu/127.17.53.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.17.53.67@KRBTEST.COM; defaulting to no policy
I20260929 20:22:25.913441 18393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:25.892141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.53.66:37609 (local address 127.17.53.126:44579)
0929 20:22:25.892360 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0929 20:22:25.892366 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:22:25.893592 (+ 1226us) server_negotiation.cc:407] Connection header received
0929 20:22:25.895908 (+ 2316us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:22:25.895911 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:22:25.895971 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:22:25.896051 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:22:25.898360 (+ 2309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:25.899158 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:25.901584 (+ 2426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:25.902348 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:25.902486 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:22:25.906167 (+ 3681us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:22:25.906176 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:22:25.906185 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:22:25.906236 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:22:25.910131 (+ 3895us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:22:25.910955 (+ 824us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:22:25.910962 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:22:25.910969 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:22:25.911147 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:22:25.911593 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:22:25.911598 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:22:25.911607 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:22:25.911880 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:22:25.912049 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0929 20:22:25.912792 (+ 743us) server_negotiation.cc:299] Negotiation successful
0929 20:22:25.913042 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 20:22:25.914685 18528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:25.892559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:44579 (local address 127.17.53.66:37609)
0929 20:22:25.893406 (+ 847us) negotiation.cc:107] Waiting for socket to connect
0929 20:22:25.893470 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0929 20:22:25.895617 (+ 2147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:22:25.896353 (+ 736us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:22:25.896366 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:22:25.897235 (+ 869us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:22:25.898153 (+ 918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:22:25.898185 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:22:25.899328 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:22:25.899337 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:22:25.901422 (+ 2085us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:22:25.901442 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:22:25.902566 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:22:25.902571 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:22:25.902787 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:22:25.903545 (+ 758us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:22:25.903567 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:22:25.905917 (+ 2350us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:22:25.910276 (+ 4359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:22:25.910284 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:22:25.910299 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:22:25.910733 (+ 434us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:22:25.911279 (+ 546us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:22:25.911282 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:22:25.911285 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:22:25.911456 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:22:25.912006 (+ 550us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:22:25.912016 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:22:25.912378 (+ 362us) client_negotiation.cc:769] Sending connection context
0929 20:22:25.912687 (+ 309us) client_negotiation.cc:240] Negotiation successful
0929 20:22:25.913268 (+ 581us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":550,"thread_start_us":325,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.53.67@KRBTEST.COM" created.
I20260929 20:22:25.916380 18526 heartbeater.cc:347] Connected to a master server at 127.17.53.126:44579
I20260929 20:22:25.916783 18526 heartbeater.cc:464] Registering TS with master...
I20260929 20:22:25.917502 18526 heartbeater.cc:511] Master 127.17.53.126:44579 requested a full tablet report, sending...
I20260929 20:22:25.919085 18200 ts_manager.cc:194] Registered new tserver with Master: ded15bfaed254bae892a2b39edf2915f (127.17.53.66:38565)
I20260929 20:22:25.919904 18200 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.53.66@KRBTEST.COM'} at 127.17.53.66:37609
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.53.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.67.keytab.
Entry for principal kudu/127.17.53.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.67.keytab.
I20260929 20:22:25.942341 17621 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
/tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790713339161492-17621-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790713339161492-17621-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790713339161492-17621-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790713339161492-17621-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.53.67:0
--local_ip_for_outbound_sockets=127.17.53.67
--webserver_interface=127.17.53.67
--webserver_port=0
--tserver_master_addrs=127.17.53.126:44579
--keytab_file=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.67.keytab
--principal=kudu/127.17.53.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.53.84:35919
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790713339161492-17621-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790713339161492-17621-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790713339161492-17621-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:22:26.266328 18531 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:22:26.266597 18531 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:22:26.266691 18531 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:22:26.273968 18531 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:22:26.274078 18531 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:22:26.274245 18531 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.53.67
I20260929 20:22:26.283921 18531 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.53.84:35919
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790713339161492-17621-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790713339161492-17621-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.67.keytab
--principal=kudu/127.17.53.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.53.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790713339161492-17621-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.17.53.67
--webserver_port=0
--tserver_master_addrs=127.17.53.126:44579
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.53.67
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790713339161492-17621-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:22:26.285244 18531 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:22:26.286548 18531 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:22:26.299224 18536 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:22:26.299228 18537 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:22:26.299384 18539 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:22:26.300513 18531 server_base.cc:1034] running on GCE node
I20260929 20:22:26.301127 18531 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:22:26.301956 18531 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:22:26.303155 18531 hybrid_clock.cc:648] HybridClock initialized: now 1790713346303106 us; error 35 us; skew 500 ppm
I20260929 20:22:26.303418 18531 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:22:26 dist-test-slave-zbnc krb5kdc[18153](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713346, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.53.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:22:26.306396 18531 init.cc:377] Logged in from keytab as kudu/127.17.53.67@KRBTEST.COM (short username kudu)
I20260929 20:22:26.307895 18531 webserver.cc:460] Webserver started at http://127.17.53.67:36361/ using document root <none> and password file <none>
I20260929 20:22:26.308579 18531 fs_manager.cc:359] Metadata directory not provided
I20260929 20:22:26.308702 18531 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:22:26.309002 18531 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:22:26.311896 18531 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790713339161492-17621-0/minicluster-data/ts-2/data/instance:
uuid: "55a6b92c33e041a886f84649bcae329f"
format_stamp: "Formatted at 2026-09-29 20:22:26 on dist-test-slave-zbnc"
server_key: "044dde6f02bc0d6f0c398d4364b089a0"
server_key_iv: "7d3044344def6a56ea2e16aec9f07a88"
server_key_version: "encryptionkey@0"
I20260929 20:22:26.312667 18531 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790713339161492-17621-0/minicluster-data/ts-2/wal/instance:
uuid: "55a6b92c33e041a886f84649bcae329f"
format_stamp: "Formatted at 2026-09-29 20:22:26 on dist-test-slave-zbnc"
server_key: "044dde6f02bc0d6f0c398d4364b089a0"
server_key_iv: "7d3044344def6a56ea2e16aec9f07a88"
server_key_version: "encryptionkey@0"
I20260929 20:22:26.317706 18531 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260929 20:22:26.321480 18546 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:22:26.322772 18531 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 20:22:26.322985 18531 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790713339161492-17621-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790713339161492-17621-0/minicluster-data/ts-2/wal
uuid: "55a6b92c33e041a886f84649bcae329f"
format_stamp: "Formatted at 2026-09-29 20:22:26 on dist-test-slave-zbnc"
server_key: "044dde6f02bc0d6f0c398d4364b089a0"
server_key_iv: "7d3044344def6a56ea2e16aec9f07a88"
server_key_version: "encryptionkey@0"
I20260929 20:22:26.323184 18531 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790713339161492-17621-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790713339161492-17621-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790713339161492-17621-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:22:26.344041 18531 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:22:26.348253 18531 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:22:26.348526 18531 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:22:26.349485 18531 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:22:26.350775 18531 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:22:26.350894 18531 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:22:26.351028 18531 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:22:26.351081 18531 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:22:26.382360 18531 rpc_server.cc:307] RPC server started. Bound to: 127.17.53.67:43557
I20260929 20:22:26.382413 18659 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.53.67:43557 every 8 connection(s)
I20260929 20:22:26.383733 18531 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790713339161492-17621-0/minicluster-data/ts-2/data/info.pb
I20260929 20:22:26.389065 17621 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu as pid 18531
I20260929 20:22:26.389269 17621 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790713339161492-17621-0/minicluster-data/ts-2/wal/instance
I20260929 20:22:26.389698 17621 external_mini_cluster.cc:1428] Setting key 2e67f445289627452613a7694e9aa38a
Sep 29 20:22:26 dist-test-slave-zbnc krb5kdc[18153](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713346, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.53.67@KRBTEST.COM for kudu/127.17.53.126@KRBTEST.COM
I20260929 20:22:26.404536 18393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:26.386312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.53.67:47621 (local address 127.17.53.126:44579)
0929 20:22:26.386594 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0929 20:22:26.386599 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:22:26.387916 (+ 1317us) server_negotiation.cc:407] Connection header received
0929 20:22:26.389856 (+ 1940us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:22:26.389862 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:22:26.389985 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:22:26.390094 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:22:26.391732 (+ 1638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:26.392601 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:26.394123 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:26.394541 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:26.394707 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:22:26.398206 (+ 3499us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:22:26.398214 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:22:26.398219 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:22:26.398262 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:22:26.401383 (+ 3121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:22:26.402097 (+ 714us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:22:26.402103 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:22:26.402107 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:22:26.402246 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:22:26.402653 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:22:26.402658 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:22:26.402661 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:22:26.402974 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:22:26.403131 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0929 20:22:26.403986 (+ 855us) server_negotiation.cc:299] Negotiation successful
0929 20:22:26.404227 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260929 20:22:26.405638 18662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:26.386846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:44579 (local address 127.17.53.67:47621)
0929 20:22:26.387746 (+ 900us) negotiation.cc:107] Waiting for socket to connect
0929 20:22:26.387789 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 20:22:26.389581 (+ 1792us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:22:26.390336 (+ 755us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:22:26.390345 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:22:26.390926 (+ 581us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:22:26.391569 (+ 643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:22:26.391587 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:22:26.392745 (+ 1158us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:22:26.392754 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:22:26.393982 (+ 1228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:22:26.393994 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:22:26.394814 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:22:26.394821 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:22:26.395065 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:22:26.395665 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:22:26.395691 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:22:26.397992 (+ 2301us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:22:26.401534 (+ 3542us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:22:26.401543 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:22:26.401560 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:22:26.401976 (+ 416us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:22:26.402352 (+ 376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:22:26.402357 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:22:26.402361 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:22:26.402541 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:22:26.403106 (+ 565us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:22:26.403113 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:22:26.403564 (+ 451us) client_negotiation.cc:769] Sending connection context
0929 20:22:26.403891 (+ 327us) client_negotiation.cc:240] Negotiation successful
0929 20:22:26.404438 (+ 547us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":526,"thread_start_us":274,"threads_started":1}
I20260929 20:22:26.406888 18660 heartbeater.cc:347] Connected to a master server at 127.17.53.126:44579
I20260929 20:22:26.407173 18660 heartbeater.cc:464] Registering TS with master...
I20260929 20:22:26.407876 18660 heartbeater.cc:511] Master 127.17.53.126:44579 requested a full tablet report, sending...
I20260929 20:22:26.409247 18200 ts_manager.cc:194] Registered new tserver with Master: 55a6b92c33e041a886f84649bcae329f (127.17.53.67:43557)
I20260929 20:22:26.410006 18200 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.53.67@KRBTEST.COM'} at 127.17.53.67:47621
I20260929 20:22:26.417035 17621 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 20:22:26.435214 18393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:26.423090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50952 (local address 127.17.53.126:44579)
0929 20:22:26.423244 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0929 20:22:26.423249 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:22:26.423741 (+ 492us) server_negotiation.cc:407] Connection header received
0929 20:22:26.423937 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:22:26.423941 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:22:26.423999 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:22:26.424119 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:22:26.425553 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:26.426283 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:26.427630 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:26.428048 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:26.428181 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:22:26.430379 (+ 2198us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:22:26.430383 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:22:26.430387 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:22:26.430416 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:22:26.432643 (+ 2227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:22:26.433285 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:22:26.433291 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:22:26.433295 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:22:26.433419 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:22:26.433778 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:22:26.433783 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:22:26.433786 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:22:26.434092 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:22:26.434343 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0929 20:22:26.434676 (+ 333us) server_negotiation.cc:299] Negotiation successful
0929 20:22:26.434928 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 20:22:26.444043 18391 heartbeater.cc:503] Master 127.17.53.126:44579 was elected leader, sending a full tablet report...
I20260929 20:22:26.441416 18200 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50952:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 20:22:26.449090 18200 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 20:22:26.479465 18675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:26.460010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.65:45613 (local address 127.0.0.1:56268)
0929 20:22:26.469206 (+ 9196us) negotiation.cc:107] Waiting for socket to connect
0929 20:22:26.469278 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0929 20:22:26.469496 (+ 218us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:22:26.472425 (+ 2929us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:22:26.472431 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:22:26.472459 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:22:26.472859 (+ 400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:22:26.472871 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:22:26.474670 (+ 1799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:22:26.474678 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:22:26.476876 (+ 2198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:22:26.476889 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:22:26.478383 (+ 1494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:22:26.478390 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:22:26.478581 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:22:26.478635 (+ 54us) client_negotiation.cc:769] Sending connection context
0929 20:22:26.478790 (+ 155us) client_negotiation.cc:240] Negotiation successful
0929 20:22:26.479015 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8851,"mutex_wait_us":83,"thread_start_us":11481,"threads_started":1}
I20260929 20:22:26.480309 18672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:26.460644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.67:43557 (local address 127.0.0.1:57988)
0929 20:22:26.471747 (+ 11103us) negotiation.cc:107] Waiting for socket to connect
0929 20:22:26.471769 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 20:22:26.471879 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:22:26.472957 (+ 1078us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:22:26.472965 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:22:26.472991 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:22:26.473367 (+ 376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:22:26.473378 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:22:26.475265 (+ 1887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:22:26.475272 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:22:26.477346 (+ 2074us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:22:26.477360 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:22:26.479468 (+ 2108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:22:26.479474 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:22:26.479648 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:22:26.479688 (+ 40us) client_negotiation.cc:769] Sending connection context
0929 20:22:26.479787 (+ 99us) client_negotiation.cc:240] Negotiation successful
0929 20:22:26.479906 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10985,"mutex_wait_us":46,"thread_start_us":8070,"threads_started":1}
I20260929 20:22:26.480610 18673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:26.460156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56268 (local address 127.17.53.65:45613)
0929 20:22:26.471475 (+ 11319us) server_negotiation.cc:206] Beginning negotiation
0929 20:22:26.471486 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 20:22:26.471521 (+ 35us) server_negotiation.cc:407] Connection header received
0929 20:22:26.471632 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:22:26.471640 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:22:26.471915 (+ 275us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:22:26.472107 (+ 192us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:22:26.473045 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:26.474552 (+ 1507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:26.477053 (+ 2501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:26.478158 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:26.478329 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:22:26.479833 (+ 1504us) server_negotiation.cc:1035] Waiting for connection context
0929 20:22:26.479904 (+ 71us) server_negotiation.cc:299] Negotiation successful
0929 20:22:26.480090 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11144,"thread_start_us":11006,"threads_started":1}
I20260929 20:22:26.481196 18677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:26.461279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57988 (local address 127.17.53.67:43557)
0929 20:22:26.470245 (+ 8966us) server_negotiation.cc:206] Beginning negotiation
0929 20:22:26.470268 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0929 20:22:26.471999 (+ 1731us) server_negotiation.cc:407] Connection header received
0929 20:22:26.472239 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:22:26.472247 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:22:26.472497 (+ 250us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:22:26.472682 (+ 185us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:22:26.473567 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:26.475098 (+ 1531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:26.477517 (+ 2419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:26.478971 (+ 1454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:26.479088 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:22:26.479313 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0929 20:22:26.480700 (+ 1387us) server_negotiation.cc:299] Negotiation successful
0929 20:22:26.480871 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8754,"thread_start_us":8582,"threads_started":1}
I20260929 20:22:26.481323 18674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:26.460372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.66:38565 (local address 127.0.0.1:33756)
0929 20:22:26.470673 (+ 10301us) negotiation.cc:107] Waiting for socket to connect
0929 20:22:26.470701 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0929 20:22:26.470869 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:22:26.473638 (+ 2769us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:22:26.473643 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:22:26.473667 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:22:26.474002 (+ 335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:22:26.474012 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:22:26.476552 (+ 2540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:22:26.476557 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:22:26.478543 (+ 1986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:22:26.478557 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:22:26.480479 (+ 1922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:22:26.480485 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:22:26.480683 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:22:26.480715 (+ 32us) client_negotiation.cc:769] Sending connection context
0929 20:22:26.480829 (+ 114us) client_negotiation.cc:240] Negotiation successful
0929 20:22:26.480999 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10115,"thread_start_us":9948,"threads_started":1}
I20260929 20:22:26.481854 18676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:26.460961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33756 (local address 127.17.53.66:38565)
0929 20:22:26.473140 (+ 12179us) server_negotiation.cc:206] Beginning negotiation
0929 20:22:26.473146 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:22:26.473167 (+ 21us) server_negotiation.cc:407] Connection header received
0929 20:22:26.473234 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:22:26.473239 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:22:26.473396 (+ 157us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:22:26.473638 (+ 242us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:22:26.474157 (+ 519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:26.476412 (+ 2255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:26.478728 (+ 2316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:26.480247 (+ 1519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:26.480415 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:22:26.480618 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0929 20:22:26.480952 (+ 334us) server_negotiation.cc:299] Negotiation successful
0929 20:22:26.481134 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12032,"thread_start_us":11926,"threads_started":1}
I20260929 20:22:26.484721 18325 tablet_service.cc:1467] Processing CreateTablet for tablet a503235ffd2346c29213b6828f409c37 (DEFAULT_TABLE table=test-table [id=8fb7a6b0e0bb46de9e29af4f527ff0d2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:22:26.484721 18594 tablet_service.cc:1467] Processing CreateTablet for tablet a503235ffd2346c29213b6828f409c37 (DEFAULT_TABLE table=test-table [id=8fb7a6b0e0bb46de9e29af4f527ff0d2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:22:26.485073 18460 tablet_service.cc:1467] Processing CreateTablet for tablet a503235ffd2346c29213b6828f409c37 (DEFAULT_TABLE table=test-table [id=8fb7a6b0e0bb46de9e29af4f527ff0d2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:22:26.486189 18460 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a503235ffd2346c29213b6828f409c37. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:22:26.486181 18325 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a503235ffd2346c29213b6828f409c37. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:22:26.486197 18594 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a503235ffd2346c29213b6828f409c37. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:22:26.497475 18678 tablet_bootstrap.cc:492] T a503235ffd2346c29213b6828f409c37 P ded15bfaed254bae892a2b39edf2915f: Bootstrap starting.
I20260929 20:22:26.500361 18679 tablet_bootstrap.cc:492] T a503235ffd2346c29213b6828f409c37 P 0da9582139ef4cfe91dbcbbe41311127: Bootstrap starting.
I20260929 20:22:26.501384 18678 tablet_bootstrap.cc:654] T a503235ffd2346c29213b6828f409c37 P ded15bfaed254bae892a2b39edf2915f: Neither blocks nor log segments found. Creating new log.
I20260929 20:22:26.502907 18678 log.cc:824] T a503235ffd2346c29213b6828f409c37 P ded15bfaed254bae892a2b39edf2915f: Log is configured to *not* fsync() on all Append() calls
I20260929 20:22:26.503602 18680 tablet_bootstrap.cc:492] T a503235ffd2346c29213b6828f409c37 P 55a6b92c33e041a886f84649bcae329f: Bootstrap starting.
I20260929 20:22:26.504823 18679 tablet_bootstrap.cc:654] T a503235ffd2346c29213b6828f409c37 P 0da9582139ef4cfe91dbcbbe41311127: Neither blocks nor log segments found. Creating new log.
I20260929 20:22:26.505925 18678 tablet_bootstrap.cc:492] T a503235ffd2346c29213b6828f409c37 P ded15bfaed254bae892a2b39edf2915f: No bootstrap required, opened a new log
I20260929 20:22:26.506067 18679 log.cc:824] T a503235ffd2346c29213b6828f409c37 P 0da9582139ef4cfe91dbcbbe41311127: Log is configured to *not* fsync() on all Append() calls
I20260929 20:22:26.506208 18678 ts_tablet_manager.cc:1397] T a503235ffd2346c29213b6828f409c37 P ded15bfaed254bae892a2b39edf2915f: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.006s
I20260929 20:22:26.507077 18680 tablet_bootstrap.cc:654] T a503235ffd2346c29213b6828f409c37 P 55a6b92c33e041a886f84649bcae329f: Neither blocks nor log segments found. Creating new log.
I20260929 20:22:26.508266 18680 log.cc:824] T a503235ffd2346c29213b6828f409c37 P 55a6b92c33e041a886f84649bcae329f: Log is configured to *not* fsync() on all Append() calls
I20260929 20:22:26.509157 18679 tablet_bootstrap.cc:492] T a503235ffd2346c29213b6828f409c37 P 0da9582139ef4cfe91dbcbbe41311127: No bootstrap required, opened a new log
I20260929 20:22:26.509408 18679 ts_tablet_manager.cc:1397] T a503235ffd2346c29213b6828f409c37 P 0da9582139ef4cfe91dbcbbe41311127: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260929 20:22:26.511204 18680 tablet_bootstrap.cc:492] T a503235ffd2346c29213b6828f409c37 P 55a6b92c33e041a886f84649bcae329f: No bootstrap required, opened a new log
I20260929 20:22:26.511444 18680 ts_tablet_manager.cc:1397] T a503235ffd2346c29213b6828f409c37 P 55a6b92c33e041a886f84649bcae329f: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260929 20:22:26.514358 18678 raft_consensus.cc:348] T a503235ffd2346c29213b6828f409c37 P ded15bfaed254bae892a2b39edf2915f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da9582139ef4cfe91dbcbbe41311127" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 45613 } } peers { permanent_uuid: "ded15bfaed254bae892a2b39edf2915f" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 38565 } } peers { permanent_uuid: "55a6b92c33e041a886f84649bcae329f" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 43557 } }
I20260929 20:22:26.514734 18678 raft_consensus.cc:374] T a503235ffd2346c29213b6828f409c37 P ded15bfaed254bae892a2b39edf2915f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:22:26.514829 18678 raft_consensus.cc:729] T a503235ffd2346c29213b6828f409c37 P ded15bfaed254bae892a2b39edf2915f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ded15bfaed254bae892a2b39edf2915f, State: Initialized, Role: FOLLOWER
I20260929 20:22:26.515733 18678 consensus_queue.cc:260] T a503235ffd2346c29213b6828f409c37 P ded15bfaed254bae892a2b39edf2915f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da9582139ef4cfe91dbcbbe41311127" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 45613 } } peers { permanent_uuid: "ded15bfaed254bae892a2b39edf2915f" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 38565 } } peers { permanent_uuid: "55a6b92c33e041a886f84649bcae329f" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 43557 } }
I20260929 20:22:26.516882 18526 heartbeater.cc:503] Master 127.17.53.126:44579 was elected leader, sending a full tablet report...
I20260929 20:22:26.517364 18678 ts_tablet_manager.cc:1428] T a503235ffd2346c29213b6828f409c37 P ded15bfaed254bae892a2b39edf2915f: Time spent starting tablet: real 0.011s user 0.003s sys 0.008s
I20260929 20:22:26.520304 18679 raft_consensus.cc:348] T a503235ffd2346c29213b6828f409c37 P 0da9582139ef4cfe91dbcbbe41311127 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da9582139ef4cfe91dbcbbe41311127" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 45613 } } peers { permanent_uuid: "ded15bfaed254bae892a2b39edf2915f" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 38565 } } peers { permanent_uuid: "55a6b92c33e041a886f84649bcae329f" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 43557 } }
I20260929 20:22:26.520828 18679 raft_consensus.cc:374] T a503235ffd2346c29213b6828f409c37 P 0da9582139ef4cfe91dbcbbe41311127 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:22:26.520987 18679 raft_consensus.cc:729] T a503235ffd2346c29213b6828f409c37 P 0da9582139ef4cfe91dbcbbe41311127 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0da9582139ef4cfe91dbcbbe41311127, State: Initialized, Role: FOLLOWER
I20260929 20:22:26.521661 18679 consensus_queue.cc:260] T a503235ffd2346c29213b6828f409c37 P 0da9582139ef4cfe91dbcbbe41311127 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da9582139ef4cfe91dbcbbe41311127" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 45613 } } peers { permanent_uuid: "ded15bfaed254bae892a2b39edf2915f" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 38565 } } peers { permanent_uuid: "55a6b92c33e041a886f84649bcae329f" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 43557 } }
I20260929 20:22:26.521577 18680 raft_consensus.cc:348] T a503235ffd2346c29213b6828f409c37 P 55a6b92c33e041a886f84649bcae329f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da9582139ef4cfe91dbcbbe41311127" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 45613 } } peers { permanent_uuid: "ded15bfaed254bae892a2b39edf2915f" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 38565 } } peers { permanent_uuid: "55a6b92c33e041a886f84649bcae329f" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 43557 } }
I20260929 20:22:26.522017 18680 raft_consensus.cc:374] T a503235ffd2346c29213b6828f409c37 P 55a6b92c33e041a886f84649bcae329f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:22:26.522135 18680 raft_consensus.cc:729] T a503235ffd2346c29213b6828f409c37 P 55a6b92c33e041a886f84649bcae329f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 55a6b92c33e041a886f84649bcae329f, State: Initialized, Role: FOLLOWER
I20260929 20:22:26.522706 18680 consensus_queue.cc:260] T a503235ffd2346c29213b6828f409c37 P 55a6b92c33e041a886f84649bcae329f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da9582139ef4cfe91dbcbbe41311127" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 45613 } } peers { permanent_uuid: "ded15bfaed254bae892a2b39edf2915f" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 38565 } } peers { permanent_uuid: "55a6b92c33e041a886f84649bcae329f" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 43557 } }
I20260929 20:22:26.524192 18679 ts_tablet_manager.cc:1428] T a503235ffd2346c29213b6828f409c37 P 0da9582139ef4cfe91dbcbbe41311127: Time spent starting tablet: real 0.015s user 0.015s sys 0.000s
I20260929 20:22:26.525269 18660 heartbeater.cc:503] Master 127.17.53.126:44579 was elected leader, sending a full tablet report...
I20260929 20:22:26.525903 18680 ts_tablet_manager.cc:1428] T a503235ffd2346c29213b6828f409c37 P 55a6b92c33e041a886f84649bcae329f: Time spent starting tablet: real 0.014s user 0.014s sys 0.000s
W20260929 20:22:26.635421 18661 tablet.cc:2369] T a503235ffd2346c29213b6828f409c37 P 55a6b92c33e041a886f84649bcae329f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 20:22:26.641531 18527 tablet.cc:2369] T a503235ffd2346c29213b6828f409c37 P ded15bfaed254bae892a2b39edf2915f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 20:22:26.662253 18392 tablet.cc:2369] T a503235ffd2346c29213b6828f409c37 P 0da9582139ef4cfe91dbcbbe41311127: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 20:22:26.747499 18685 raft_consensus.cc:482] T a503235ffd2346c29213b6828f409c37 P 0da9582139ef4cfe91dbcbbe41311127 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 20:22:26.747850 18685 raft_consensus.cc:504] T a503235ffd2346c29213b6828f409c37 P 0da9582139ef4cfe91dbcbbe41311127 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da9582139ef4cfe91dbcbbe41311127" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 45613 } } peers { permanent_uuid: "ded15bfaed254bae892a2b39edf2915f" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 38565 } } peers { permanent_uuid: "55a6b92c33e041a886f84649bcae329f" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 43557 } }
I20260929 20:22:26.749508 18685 leader_election.cc:288] T a503235ffd2346c29213b6828f409c37 P 0da9582139ef4cfe91dbcbbe41311127 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ded15bfaed254bae892a2b39edf2915f (127.17.53.66:38565), 55a6b92c33e041a886f84649bcae329f (127.17.53.67:43557)
I20260929 20:22:26.757885 18677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:26.749960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.53.65:52523 (local address 127.17.53.67:43557)
0929 20:22:26.750162 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0929 20:22:26.750171 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 20:22:26.751472 (+ 1301us) server_negotiation.cc:407] Connection header received
0929 20:22:26.751620 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:22:26.751623 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:22:26.751679 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:22:26.751750 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:22:26.752305 (+ 555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:26.753249 (+ 944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:26.755601 (+ 2352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:26.756819 (+ 1218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:26.756975 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:22:26.757049 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0929 20:22:26.757498 (+ 449us) server_negotiation.cc:299] Negotiation successful
0929 20:22:26.757628 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260929 20:22:26.757885 18676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:26.750084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.53.65:42633 (local address 127.17.53.66:38565)
0929 20:22:26.750301 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0929 20:22:26.750309 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 20:22:26.750895 (+ 586us) server_negotiation.cc:407] Connection header received
0929 20:22:26.751154 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:22:26.751159 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:22:26.751243 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:22:26.751347 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:22:26.751943 (+ 596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:26.752943 (+ 1000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:26.755587 (+ 2644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:26.756818 (+ 1231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:26.756975 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:22:26.757049 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0929 20:22:26.757498 (+ 449us) server_negotiation.cc:299] Negotiation successful
0929 20:22:26.757624 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260929 20:22:26.758118 18688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:26.750065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.67:43557 (local address 127.17.53.65:52523)
0929 20:22:26.751360 (+ 1295us) negotiation.cc:107] Waiting for socket to connect
0929 20:22:26.751376 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 20:22:26.751490 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:22:26.751802 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:22:26.751806 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:22:26.751828 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:22:26.752138 (+ 310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:22:26.752145 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:22:26.753388 (+ 1243us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:22:26.753393 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:22:26.755465 (+ 2072us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:22:26.755478 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:22:26.757074 (+ 1596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:22:26.757082 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:22:26.757308 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:22:26.757363 (+ 55us) client_negotiation.cc:769] Sending connection context
0929 20:22:26.757461 (+ 98us) client_negotiation.cc:240] Negotiation successful
0929 20:22:26.757664 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1180,"mutex_wait_us":91,"thread_start_us":975,"threads_started":1}
I20260929 20:22:26.758158 18687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:26.749989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.66:38565 (local address 127.17.53.65:42633)
0929 20:22:26.750779 (+ 790us) negotiation.cc:107] Waiting for socket to connect
0929 20:22:26.750795 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 20:22:26.750997 (+ 202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:22:26.751394 (+ 397us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:22:26.751402 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:22:26.751434 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:22:26.751807 (+ 373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:22:26.751817 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:22:26.753067 (+ 1250us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:22:26.753070 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:22:26.755409 (+ 2339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:22:26.755430 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:22:26.757091 (+ 1661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:22:26.757095 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:22:26.757314 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:22:26.757367 (+ 53us) client_negotiation.cc:769] Sending connection context
0929 20:22:26.757461 (+ 94us) client_negotiation.cc:240] Negotiation successful
0929 20:22:26.757664 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":658,"mutex_wait_us":92,"thread_start_us":474,"threads_started":1}
I20260929 20:22:26.759193 18615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a503235ffd2346c29213b6828f409c37" candidate_uuid: "0da9582139ef4cfe91dbcbbe41311127" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "55a6b92c33e041a886f84649bcae329f" is_pre_election: true
I20260929 20:22:26.759783 18615 raft_consensus.cc:2454] T a503235ffd2346c29213b6828f409c37 P 55a6b92c33e041a886f84649bcae329f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da9582139ef4cfe91dbcbbe41311127 in term 0.
I20260929 20:22:26.760048 18481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a503235ffd2346c29213b6828f409c37" candidate_uuid: "0da9582139ef4cfe91dbcbbe41311127" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ded15bfaed254bae892a2b39edf2915f" is_pre_election: true
I20260929 20:22:26.760497 18278 leader_election.cc:302] T a503235ffd2346c29213b6828f409c37 P 0da9582139ef4cfe91dbcbbe41311127 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da9582139ef4cfe91dbcbbe41311127, 55a6b92c33e041a886f84649bcae329f; no voters:
I20260929 20:22:26.760603 18481 raft_consensus.cc:2454] T a503235ffd2346c29213b6828f409c37 P ded15bfaed254bae892a2b39edf2915f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0da9582139ef4cfe91dbcbbe41311127 in term 0.
I20260929 20:22:26.760942 18685 raft_consensus.cc:2781] T a503235ffd2346c29213b6828f409c37 P 0da9582139ef4cfe91dbcbbe41311127 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 20:22:26.761114 18685 raft_consensus.cc:482] T a503235ffd2346c29213b6828f409c37 P 0da9582139ef4cfe91dbcbbe41311127 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 20:22:26.761252 18685 raft_consensus.cc:3037] T a503235ffd2346c29213b6828f409c37 P 0da9582139ef4cfe91dbcbbe41311127 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:22:26.763551 18685 raft_consensus.cc:504] T a503235ffd2346c29213b6828f409c37 P 0da9582139ef4cfe91dbcbbe41311127 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da9582139ef4cfe91dbcbbe41311127" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 45613 } } peers { permanent_uuid: "ded15bfaed254bae892a2b39edf2915f" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 38565 } } peers { permanent_uuid: "55a6b92c33e041a886f84649bcae329f" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 43557 } }
I20260929 20:22:26.764184 18685 leader_election.cc:288] T a503235ffd2346c29213b6828f409c37 P 0da9582139ef4cfe91dbcbbe41311127 [CANDIDATE]: Term 1 election: Requested vote from peers ded15bfaed254bae892a2b39edf2915f (127.17.53.66:38565), 55a6b92c33e041a886f84649bcae329f (127.17.53.67:43557)
I20260929 20:22:26.764581 18481 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a503235ffd2346c29213b6828f409c37" candidate_uuid: "0da9582139ef4cfe91dbcbbe41311127" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ded15bfaed254bae892a2b39edf2915f"
I20260929 20:22:26.764786 18481 raft_consensus.cc:3037] T a503235ffd2346c29213b6828f409c37 P ded15bfaed254bae892a2b39edf2915f [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:22:26.764856 18615 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a503235ffd2346c29213b6828f409c37" candidate_uuid: "0da9582139ef4cfe91dbcbbe41311127" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "55a6b92c33e041a886f84649bcae329f"
I20260929 20:22:26.765054 18615 raft_consensus.cc:3037] T a503235ffd2346c29213b6828f409c37 P 55a6b92c33e041a886f84649bcae329f [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:22:26.768083 18481 raft_consensus.cc:2454] T a503235ffd2346c29213b6828f409c37 P ded15bfaed254bae892a2b39edf2915f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da9582139ef4cfe91dbcbbe41311127 in term 1.
I20260929 20:22:26.768172 18615 raft_consensus.cc:2454] T a503235ffd2346c29213b6828f409c37 P 55a6b92c33e041a886f84649bcae329f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0da9582139ef4cfe91dbcbbe41311127 in term 1.
I20260929 20:22:26.768543 18278 leader_election.cc:302] T a503235ffd2346c29213b6828f409c37 P 0da9582139ef4cfe91dbcbbe41311127 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0da9582139ef4cfe91dbcbbe41311127, 55a6b92c33e041a886f84649bcae329f; no voters:
I20260929 20:22:26.768831 18685 raft_consensus.cc:2781] T a503235ffd2346c29213b6828f409c37 P 0da9582139ef4cfe91dbcbbe41311127 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:22:26.769269 18685 raft_consensus.cc:686] T a503235ffd2346c29213b6828f409c37 P 0da9582139ef4cfe91dbcbbe41311127 [term 1 LEADER]: Becoming Leader. State: Replica: 0da9582139ef4cfe91dbcbbe41311127, State: Running, Role: LEADER
I20260929 20:22:26.769831 18685 consensus_queue.cc:237] T a503235ffd2346c29213b6828f409c37 P 0da9582139ef4cfe91dbcbbe41311127 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da9582139ef4cfe91dbcbbe41311127" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 45613 } } peers { permanent_uuid: "ded15bfaed254bae892a2b39edf2915f" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 38565 } } peers { permanent_uuid: "55a6b92c33e041a886f84649bcae329f" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 43557 } }
I20260929 20:22:26.774186 18199 catalog_manager.cc:5495] T a503235ffd2346c29213b6828f409c37 P 0da9582139ef4cfe91dbcbbe41311127 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0da9582139ef4cfe91dbcbbe41311127 (127.17.53.65). New cstate: current_term: 1 leader_uuid: "0da9582139ef4cfe91dbcbbe41311127" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0da9582139ef4cfe91dbcbbe41311127" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 45613 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ded15bfaed254bae892a2b39edf2915f" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 38565 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "55a6b92c33e041a886f84649bcae329f" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 43557 } health_report { overall_health: UNKNOWN } } }
Sep 29 20:22:26 dist-test-slave-zbnc krb5kdc[18153](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713344, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.53.65@KRBTEST.COM
I20260929 20:22:26.852456 18673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:26.837974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56278 (local address 127.17.53.65:45613)
0929 20:22:26.838222 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0929 20:22:26.838231 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 20:22:26.838652 (+ 421us) server_negotiation.cc:407] Connection header received
0929 20:22:26.838917 (+ 265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:22:26.838924 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:22:26.839026 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:22:26.839122 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:22:26.840705 (+ 1583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:26.841412 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:26.842579 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:26.843044 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:26.843169 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:22:26.847152 (+ 3983us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:22:26.847166 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:22:26.847181 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:22:26.847217 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:22:26.849792 (+ 2575us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:22:26.850422 (+ 630us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:22:26.850436 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:22:26.850438 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:22:26.850586 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:22:26.850968 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:22:26.850973 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:22:26.850976 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:22:26.851317 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:22:26.851610 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0929 20:22:26.851893 (+ 283us) server_negotiation.cc:299] Negotiation successful
0929 20:22:26.852141 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
Sep 29 20:22:26 dist-test-slave-zbnc krb5kdc[18153](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713346, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 29 20:22:26 dist-test-slave-zbnc krb5kdc[18153](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713346, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.17.53.65@KRBTEST.COM
I20260929 20:22:26.898033 18673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:26.883835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56292 (local address 127.17.53.65:45613)
0929 20:22:26.884082 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0929 20:22:26.884089 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:22:26.884365 (+ 276us) server_negotiation.cc:407] Connection header received
0929 20:22:26.884579 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:22:26.884583 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:22:26.884646 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:22:26.884751 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:22:26.886134 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:26.887044 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:26.888415 (+ 1371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:26.888836 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:26.888966 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:22:26.892883 (+ 3917us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:22:26.892887 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:22:26.892891 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:22:26.892923 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:22:26.895577 (+ 2654us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:22:26.896176 (+ 599us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:22:26.896185 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:22:26.896189 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:22:26.896316 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:22:26.896687 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:22:26.896692 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:22:26.896696 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:22:26.896950 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:22:26.897247 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0929 20:22:26.897481 (+ 234us) server_negotiation.cc:299] Negotiation successful
0929 20:22:26.897697 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
Sep 29 20:22:26 dist-test-slave-zbnc krb5kdc[18153](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713346, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 29 20:22:26 dist-test-slave-zbnc krb5kdc[18153](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713346, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.53.65@KRBTEST.COM
I20260929 20:22:26.944316 18673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:26.930176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56294 (local address 127.17.53.65:45613)
0929 20:22:26.930375 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0929 20:22:26.930383 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 20:22:26.930734 (+ 351us) server_negotiation.cc:407] Connection header received
0929 20:22:26.930999 (+ 265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:22:26.931003 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:22:26.931079 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:22:26.931181 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:22:26.932507 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:26.933262 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:26.934504 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:26.934960 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:26.935085 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:22:26.938979 (+ 3894us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:22:26.938984 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:22:26.938987 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:22:26.939022 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:22:26.941872 (+ 2850us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:22:26.942525 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:22:26.942529 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:22:26.942532 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:22:26.942675 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:22:26.943026 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:22:26.943029 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:22:26.943037 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:22:26.943305 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:22:26.943492 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0929 20:22:26.943716 (+ 224us) server_negotiation.cc:299] Negotiation successful
0929 20:22:26.943877 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
W20260929 20:22:26.945971 18305 tablet_service.cc:2908] Rejecting scan request for tablet a503235ffd2346c29213b6828f409c37: Uninitialized: clean time has not yet been initialized
I20260929 20:22:26.948799 17621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu pid 18262
I20260929 20:22:26.961318 18673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:26.949654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56302 (local address 127.17.53.65:45613)
0929 20:22:26.949803 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0929 20:22:26.949806 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0929 20:22:26.950226 (+ 420us) server_negotiation.cc:407] Connection header received
0929 20:22:26.950360 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:22:26.950363 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:22:26.950415 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:22:26.950485 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:22:26.951714 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:26.952427 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:26.953544 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:26.953933 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:26.954021 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:22:26.956116 (+ 2095us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:22:26.956121 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:22:26.956124 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:22:26.956151 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:22:26.958591 (+ 2440us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:22:26.959358 (+ 767us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:22:26.959363 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:22:26.959365 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:22:26.959668 (+ 303us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:22:26.960045 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:22:26.960050 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:22:26.960058 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:22:26.960407 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:22:26.960603 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0929 20:22:26.960823 (+ 220us) server_negotiation.cc:299] Negotiation successful
0929 20:22:26.961024 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260929 20:22:26.962040 18386 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:22:27.054776 17621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu with pid 18262
I20260929 20:22:27.076136 17621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu pid 18397
Sep 29 20:22:27 dist-test-slave-zbnc krb5kdc[18153](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713346, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.53.66@KRBTEST.COM
I20260929 20:22:27.090777 18676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:27.076878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33758 (local address 127.17.53.66:38565)
0929 20:22:27.077050 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0929 20:22:27.077058 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 20:22:27.077156 (+ 98us) server_negotiation.cc:407] Connection header received
0929 20:22:27.077346 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:22:27.077350 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:22:27.077412 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:22:27.077514 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:22:27.078742 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:27.079558 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:27.080876 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:27.081273 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:27.081401 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:22:27.085379 (+ 3978us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:22:27.085393 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:22:27.085409 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:22:27.085446 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:22:27.088001 (+ 2555us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:22:27.088672 (+ 671us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:22:27.088681 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:22:27.088683 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:22:27.088814 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:22:27.089170 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:22:27.089175 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:22:27.089183 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:22:27.089528 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:22:27.089822 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0929 20:22:27.090164 (+ 342us) server_negotiation.cc:299] Negotiation successful
0929 20:22:27.090379 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 20:22:27.091580 18521 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:22:27.178929 17621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu with pid 18397
I20260929 20:22:27.196496 17621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu pid 18531
Sep 29 20:22:27 dist-test-slave-zbnc krb5kdc[18153](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713346, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.53.67@KRBTEST.COM
I20260929 20:22:27.210988 18677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:27.197251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57996 (local address 127.17.53.67:43557)
0929 20:22:27.197431 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0929 20:22:27.197436 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:22:27.197457 (+ 21us) server_negotiation.cc:407] Connection header received
0929 20:22:27.197623 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:22:27.197629 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:22:27.197715 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:22:27.197807 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:22:27.199104 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:27.199885 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:27.201133 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:27.201535 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:27.201635 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:22:27.205652 (+ 4017us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:22:27.205669 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:22:27.205686 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:22:27.205719 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:22:27.208645 (+ 2926us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:22:27.209231 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:22:27.209239 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:22:27.209243 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:22:27.209391 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:22:27.209703 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:22:27.209706 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:22:27.209708 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:22:27.210072 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:22:27.210299 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0929 20:22:27.210484 (+ 185us) server_negotiation.cc:299] Negotiation successful
0929 20:22:27.210654 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260929 20:22:27.211701 18655 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:22:27.299399 17621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu with pid 18531
I20260929 20:22:27.315668 17621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu pid 18167
I20260929 20:22:27.316769 18230 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:22:27.424846 17621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu with pid 18167
2026-09-29T20:22:27Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (4308 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790713339161492-17621-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 20:22:27 dist-test-slave-zbnc krb5kdc[18708](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 20:22:27 dist-test-slave-zbnc krb5kdc[18708](info): set up 2 sockets
Sep 29 20:22:27 dist-test-slave-zbnc krb5kdc[18708](info): commencing operation
krb5kdc: starting...
W20260929 20:22:28.512369 17621 mini_kdc.cc:121] Time spent starting KDC: real 1.045s user 0.003s sys 0.012s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790713339161492-17621-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790713339161492-17621-0/krb5kdc/test-user.keytab.
Sep 29 20:22:28 dist-test-slave-zbnc krb5kdc[18708](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713348, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T20:22:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T20:22:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.17.53.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.53.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.53.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.126.keytab.
Entry for principal kudu/127.17.53.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.126.keytab.
I20260929 20:22:28.696499 17621 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
/tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790713339161492-17621-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790713339161492-17621-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790713339161492-17621-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790713339161492-17621-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.53.126:43983
--webserver_interface=127.17.53.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.126.keytab
--principal=kudu/127.17.53.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.53.84:34623
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.53.126:43983
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790713339161492-17621-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790713339161492-17621-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790713339161492-17621-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:22:29.018453 18721 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:22:29.018869 18721 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:22:29.019068 18721 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:22:29.026355 18721 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:22:29.026501 18721 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260929 20:22:29.026598 18721 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:22:29.026719 18721 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:22:29.026801 18721 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:22:29.026908 18721 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:22:29.036878 18721 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.53.84:34623
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790713339161492-17621-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790713339161492-17621-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.53.126:43983
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.126.keytab
--principal=kudu/127.17.53.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.53.126:43983
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790713339161492-17621-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.53.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790713339161492-17621-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:22:29.038216 18721 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:22:29.039580 18721 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:22:29.051357 18726 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:22:29.051641 18727 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:22:29.051352 18729 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:22:29.052737 18721 server_base.cc:1034] running on GCE node
I20260929 20:22:29.053767 18721 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:22:29.055794 18721 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:22:29.057103 18721 hybrid_clock.cc:648] HybridClock initialized: now 1790713349056977 us; error 95 us; skew 500 ppm
I20260929 20:22:29.057576 18721 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:22:29 dist-test-slave-zbnc krb5kdc[18708](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713349, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.53.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:22:29.061443 18721 init.cc:377] Logged in from keytab as kudu/127.17.53.126@KRBTEST.COM (short username kudu)
I20260929 20:22:29.063184 18721 webserver.cc:460] Webserver started at http://127.17.53.126:44567/ using document root <none> and password file <none>
I20260929 20:22:29.063966 18721 fs_manager.cc:359] Metadata directory not provided
I20260929 20:22:29.064111 18721 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:22:29.064461 18721 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:22:29.067682 18721 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790713339161492-17621-0/minicluster-data/master-0/data/instance:
uuid: "edc009a06883495fbef928dda0915f0d"
format_stamp: "Formatted at 2026-09-29 20:22:29 on dist-test-slave-zbnc"
server_key: "7d0347d3952841e670705b705f973bb4"
server_key_iv: "f4a93bf78d8a51f26189dd66eafb53dd"
server_key_version: "encryptionkey@0"
I20260929 20:22:29.068483 18721 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790713339161492-17621-0/minicluster-data/master-0/wal/instance:
uuid: "edc009a06883495fbef928dda0915f0d"
format_stamp: "Formatted at 2026-09-29 20:22:29 on dist-test-slave-zbnc"
server_key: "7d0347d3952841e670705b705f973bb4"
server_key_iv: "f4a93bf78d8a51f26189dd66eafb53dd"
server_key_version: "encryptionkey@0"
I20260929 20:22:29.074132 18721 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260929 20:22:29.077950 18736 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:22:29.079419 18721 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 20:22:29.079602 18721 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790713339161492-17621-0/minicluster-data/master-0/data,/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790713339161492-17621-0/minicluster-data/master-0/wal
uuid: "edc009a06883495fbef928dda0915f0d"
format_stamp: "Formatted at 2026-09-29 20:22:29 on dist-test-slave-zbnc"
server_key: "7d0347d3952841e670705b705f973bb4"
server_key_iv: "f4a93bf78d8a51f26189dd66eafb53dd"
server_key_version: "encryptionkey@0"
I20260929 20:22:29.079831 18721 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790713339161492-17621-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790713339161492-17621-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790713339161492-17621-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:22:29.102375 18721 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:22:29.106835 18721 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:22:29.107153 18721 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:22:29.128602 18721 rpc_server.cc:307] RPC server started. Bound to: 127.17.53.126:43983
I20260929 20:22:29.128621 18798 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.53.126:43983 every 8 connection(s)
I20260929 20:22:29.130059 18721 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790713339161492-17621-0/minicluster-data/master-0/data/info.pb
I20260929 20:22:29.133613 17621 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu as pid 18721
I20260929 20:22:29.133858 17621 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790713339161492-17621-0/minicluster-data/master-0/wal/instance
I20260929 20:22:29.134150 18799 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:22:29.134721 17621 external_mini_cluster.cc:1428] Setting key 57296df9bf026bcc5a5a715a75bd119e
I20260929 20:22:29.146512 18799 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P edc009a06883495fbef928dda0915f0d: Bootstrap starting.
Sep 29 20:22:29 dist-test-slave-zbnc krb5kdc[18708](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713348, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.53.126@KRBTEST.COM
I20260929 20:22:29.150157 18799 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P edc009a06883495fbef928dda0915f0d: Neither blocks nor log segments found. Creating new log.
I20260929 20:22:29.151299 18799 log.cc:824] T 00000000000000000000000000000000 P edc009a06883495fbef928dda0915f0d: Log is configured to *not* fsync() on all Append() calls
I20260929 20:22:29.154532 18799 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P edc009a06883495fbef928dda0915f0d: No bootstrap required, opened a new log
I20260929 20:22:29.156911 18802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:29.136855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59600 (local address 127.17.53.126:43983)
0929 20:22:29.137582 (+ 727us) server_negotiation.cc:206] Beginning negotiation
0929 20:22:29.137594 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 20:22:29.137635 (+ 41us) server_negotiation.cc:407] Connection header received
0929 20:22:29.139211 (+ 1576us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:22:29.139231 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:22:29.139614 (+ 383us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:22:29.140014 (+ 400us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:22:29.141397 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:29.142638 (+ 1241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:29.143885 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:29.144585 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:29.144805 (+ 220us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:22:29.149273 (+ 4468us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:22:29.149281 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:22:29.149295 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:22:29.149333 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:22:29.152840 (+ 3507us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:22:29.153672 (+ 832us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:22:29.153678 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:22:29.153684 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:22:29.153846 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:22:29.154227 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:22:29.154230 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:22:29.154232 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:22:29.154776 (+ 544us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:22:29.155067 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0929 20:22:29.155474 (+ 407us) server_negotiation.cc:299] Negotiation successful
0929 20:22:29.155809 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":528,"thread_start_us":318,"threads_started":1}
I20260929 20:22:29.162891 18799 raft_consensus.cc:348] T 00000000000000000000000000000000 P edc009a06883495fbef928dda0915f0d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "edc009a06883495fbef928dda0915f0d" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 43983 } }
I20260929 20:22:29.163295 18799 raft_consensus.cc:374] T 00000000000000000000000000000000 P edc009a06883495fbef928dda0915f0d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:22:29.163411 18799 raft_consensus.cc:729] T 00000000000000000000000000000000 P edc009a06883495fbef928dda0915f0d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: edc009a06883495fbef928dda0915f0d, State: Initialized, Role: FOLLOWER
I20260929 20:22:29.163997 18799 consensus_queue.cc:260] T 00000000000000000000000000000000 P edc009a06883495fbef928dda0915f0d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "edc009a06883495fbef928dda0915f0d" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 43983 } }
I20260929 20:22:29.164222 18799 raft_consensus.cc:388] T 00000000000000000000000000000000 P edc009a06883495fbef928dda0915f0d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 20:22:29.164356 18799 raft_consensus.cc:482] T 00000000000000000000000000000000 P edc009a06883495fbef928dda0915f0d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 20:22:29.164511 18799 raft_consensus.cc:3037] T 00000000000000000000000000000000 P edc009a06883495fbef928dda0915f0d [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:22:29.166738 18799 raft_consensus.cc:504] T 00000000000000000000000000000000 P edc009a06883495fbef928dda0915f0d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "edc009a06883495fbef928dda0915f0d" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 43983 } }
I20260929 20:22:29.167232 18799 leader_election.cc:302] T 00000000000000000000000000000000 P edc009a06883495fbef928dda0915f0d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: edc009a06883495fbef928dda0915f0d; no voters:
I20260929 20:22:29.167838 18799 leader_election.cc:288] T 00000000000000000000000000000000 P edc009a06883495fbef928dda0915f0d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 20:22:29.168015 18804 raft_consensus.cc:2781] T 00000000000000000000000000000000 P edc009a06883495fbef928dda0915f0d [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:22:29.168826 18804 raft_consensus.cc:686] T 00000000000000000000000000000000 P edc009a06883495fbef928dda0915f0d [term 1 LEADER]: Becoming Leader. State: Replica: edc009a06883495fbef928dda0915f0d, State: Running, Role: LEADER
I20260929 20:22:29.169440 18804 consensus_queue.cc:237] T 00000000000000000000000000000000 P edc009a06883495fbef928dda0915f0d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "edc009a06883495fbef928dda0915f0d" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 43983 } }
I20260929 20:22:29.170003 18799 sys_catalog.cc:564] T 00000000000000000000000000000000 P edc009a06883495fbef928dda0915f0d [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:22:29.176770 18805 sys_catalog.cc:455] T 00000000000000000000000000000000 P edc009a06883495fbef928dda0915f0d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "edc009a06883495fbef928dda0915f0d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "edc009a06883495fbef928dda0915f0d" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 43983 } } }
I20260929 20:22:29.177186 18805 sys_catalog.cc:458] T 00000000000000000000000000000000 P edc009a06883495fbef928dda0915f0d [sys.catalog]: This master's current role is: LEADER
I20260929 20:22:29.178311 18806 sys_catalog.cc:455] T 00000000000000000000000000000000 P edc009a06883495fbef928dda0915f0d [sys.catalog]: SysCatalogTable state changed. Reason: New leader edc009a06883495fbef928dda0915f0d. Latest consensus state: current_term: 1 leader_uuid: "edc009a06883495fbef928dda0915f0d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "edc009a06883495fbef928dda0915f0d" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 43983 } } }
I20260929 20:22:29.178692 18806 sys_catalog.cc:458] T 00000000000000000000000000000000 P edc009a06883495fbef928dda0915f0d [sys.catalog]: This master's current role is: LEADER
I20260929 20:22:29.181105 18815 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:22:29.186542 18815 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:22:29.195864 18815 catalog_manager.cc:1347] Generated new cluster ID: f3066caec31f44dc8da2c08b6424031f
I20260929 20:22:29.196084 18815 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:22:29.239789 18815 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 20:22:29.241950 18815 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:22:29.258355 18815 catalog_manager.cc:5868] T 00000000000000000000000000000000 P edc009a06883495fbef928dda0915f0d: Generated new TSK 0
I20260929 20:22:29.259510 18815 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.17.53.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.53.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.53.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.65.keytab.
Entry for principal kudu/127.17.53.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.65.keytab.
I20260929 20:22:29.318938 17621 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
/tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790713339161492-17621-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790713339161492-17621-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790713339161492-17621-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790713339161492-17621-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.53.65:0
--local_ip_for_outbound_sockets=127.17.53.65
--webserver_interface=127.17.53.65
--webserver_port=0
--tserver_master_addrs=127.17.53.126:43983
--keytab_file=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.65.keytab
--principal=kudu/127.17.53.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.53.84:34623
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790713339161492-17621-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790713339161492-17621-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790713339161492-17621-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:22:29.664831 18826 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:22:29.665119 18826 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:22:29.665297 18826 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:22:29.672859 18826 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:22:29.672993 18826 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:22:29.673139 18826 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 20:22:29.673264 18826 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.53.65
I20260929 20:22:29.683553 18826 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.53.84:34623
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790713339161492-17621-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790713339161492-17621-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.65.keytab
--principal=kudu/127.17.53.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.53.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790713339161492-17621-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.17.53.65
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.17.53.126:43983
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.53.65
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790713339161492-17621-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:22:29.684840 18826 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:22:29.686138 18826 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:22:29.699098 18834 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:22:29.699123 18831 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:22:29.699191 18832 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:22:29.700138 18826 server_base.cc:1034] running on GCE node
I20260929 20:22:29.700745 18826 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:22:29.701622 18826 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:22:29.702901 18826 hybrid_clock.cc:648] HybridClock initialized: now 1790713349702798 us; error 35 us; skew 500 ppm
I20260929 20:22:29.703204 18826 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:22:29 dist-test-slave-zbnc krb5kdc[18708](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713349, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.53.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:22:29.706235 18826 init.cc:377] Logged in from keytab as kudu/127.17.53.65@KRBTEST.COM (short username kudu)
I20260929 20:22:29.707741 18826 webserver.cc:460] Webserver started at http://127.17.53.65:32865/ using document root <none> and password file <none>
I20260929 20:22:29.708417 18826 fs_manager.cc:359] Metadata directory not provided
I20260929 20:22:29.708541 18826 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:22:29.708856 18826 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:22:29.711745 18826 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790713339161492-17621-0/minicluster-data/ts-0/data/instance:
uuid: "744ee3020a694d7fad3c0ee1c7e50a26"
format_stamp: "Formatted at 2026-09-29 20:22:29 on dist-test-slave-zbnc"
server_key: "7ec0226698957366163945468b519b69"
server_key_iv: "a761c79b8f70b8bb4631c359d85b3509"
server_key_version: "encryptionkey@0"
I20260929 20:22:29.712503 18826 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790713339161492-17621-0/minicluster-data/ts-0/wal/instance:
uuid: "744ee3020a694d7fad3c0ee1c7e50a26"
format_stamp: "Formatted at 2026-09-29 20:22:29 on dist-test-slave-zbnc"
server_key: "7ec0226698957366163945468b519b69"
server_key_iv: "a761c79b8f70b8bb4631c359d85b3509"
server_key_version: "encryptionkey@0"
I20260929 20:22:29.717442 18826 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 20:22:29.721097 18841 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:22:29.722370 18826 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 20:22:29.722530 18826 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790713339161492-17621-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790713339161492-17621-0/minicluster-data/ts-0/wal
uuid: "744ee3020a694d7fad3c0ee1c7e50a26"
format_stamp: "Formatted at 2026-09-29 20:22:29 on dist-test-slave-zbnc"
server_key: "7ec0226698957366163945468b519b69"
server_key_iv: "a761c79b8f70b8bb4631c359d85b3509"
server_key_version: "encryptionkey@0"
I20260929 20:22:29.722807 18826 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790713339161492-17621-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790713339161492-17621-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790713339161492-17621-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:22:29.742951 18826 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:22:29.746991 18826 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:22:29.747272 18826 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:22:29.750049 18826 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:22:29.750156 18826 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:22:29.750315 18826 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:22:29.750429 18826 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
Sep 29 20:22:29 dist-test-slave-zbnc krb5kdc[18708](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713349, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.53.65@KRBTEST.COM for kudu/127.17.53.126@KRBTEST.COM
I20260929 20:22:29.781415 18850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:29.750545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.53.65:45133 (local address 127.17.53.126:43983)
0929 20:22:29.751494 (+ 949us) server_negotiation.cc:206] Beginning negotiation
0929 20:22:29.751498 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:22:29.752471 (+ 973us) server_negotiation.cc:407] Connection header received
0929 20:22:29.754526 (+ 2055us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:22:29.754532 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:22:29.754620 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:22:29.754743 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:22:29.757049 (+ 2306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:29.758128 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:29.760792 (+ 2664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:29.761391 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:29.761528 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:22:29.774949 (+ 13421us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:22:29.774954 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:22:29.774958 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:22:29.774994 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:22:29.778204 (+ 3210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:22:29.779001 (+ 797us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:22:29.779007 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:22:29.779015 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:22:29.779128 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:22:29.779610 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:22:29.779615 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:22:29.779618 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:22:29.779893 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:22:29.780044 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0929 20:22:29.780818 (+ 774us) server_negotiation.cc:299] Negotiation successful
0929 20:22:29.781036 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":840,"thread_start_us":731,"threads_started":1}
I20260929 20:22:29.782814 18851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:29.751366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:43983 (local address 127.17.53.65:45133)
0929 20:22:29.752298 (+ 932us) negotiation.cc:107] Waiting for socket to connect
0929 20:22:29.752347 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0929 20:22:29.754239 (+ 1892us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:22:29.755003 (+ 764us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:22:29.755012 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:22:29.755832 (+ 820us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:22:29.756842 (+ 1010us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:22:29.756874 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:22:29.758299 (+ 1425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:22:29.758307 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:22:29.760640 (+ 2333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:22:29.760653 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:22:29.761632 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:22:29.761639 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:22:29.761969 (+ 330us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:22:29.771726 (+ 9757us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:22:29.771753 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:22:29.774619 (+ 2866us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:22:29.778359 (+ 3740us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:22:29.778370 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:22:29.778388 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:22:29.778837 (+ 449us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:22:29.779241 (+ 404us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:22:29.779245 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:22:29.779249 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:22:29.779487 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:22:29.780020 (+ 533us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:22:29.780028 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:22:29.780410 (+ 382us) client_negotiation.cc:769] Sending connection context
0929 20:22:29.780723 (+ 313us) client_negotiation.cc:240] Negotiation successful
0929 20:22:29.781280 (+ 557us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":661,"thread_start_us":385,"threads_started":1}
I20260929 20:22:29.804081 18826 rpc_server.cc:307] RPC server started. Bound to: 127.17.53.65:37593
I20260929 20:22:29.804046 18963 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.53.65:37593 every 8 connection(s)
I20260929 20:22:29.805181 18938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:29.789222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:43983 (local address 127.17.53.65:46925)
0929 20:22:29.791528 (+ 2306us) negotiation.cc:107] Waiting for socket to connect
0929 20:22:29.791549 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 20:22:29.791698 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:22:29.792007 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:22:29.792011 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:22:29.792465 (+ 454us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:22:29.792739 (+ 274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:22:29.792748 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:22:29.793711 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:22:29.793717 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:22:29.794711 (+ 994us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:22:29.794721 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:22:29.795609 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:22:29.795619 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:22:29.795880 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:22:29.796681 (+ 801us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:22:29.796711 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:22:29.797934 (+ 1223us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:22:29.802230 (+ 4296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:22:29.802234 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:22:29.802237 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:22:29.802603 (+ 366us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:22:29.803091 (+ 488us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:22:29.803098 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:22:29.803103 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:22:29.803294 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:22:29.804019 (+ 725us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:22:29.804031 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:22:29.804268 (+ 237us) client_negotiation.cc:769] Sending connection context
0929 20:22:29.804412 (+ 144us) client_negotiation.cc:240] Negotiation successful
0929 20:22:29.804727 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2155,"thread_start_us":1999,"threads_started":1}
I20260929 20:22:29.805780 18826 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790713339161492-17621-0/minicluster-data/ts-0/data/info.pb
I20260929 20:22:29.806149 18850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:29.789216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.53.65:46925 (local address 127.17.53.126:43983)
0929 20:22:29.789428 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0929 20:22:29.789436 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 20:22:29.791628 (+ 2192us) server_negotiation.cc:407] Connection header received
0929 20:22:29.791817 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:22:29.791826 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:22:29.791888 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:22:29.791985 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:22:29.792875 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:29.793578 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:29.794846 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:29.795355 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:29.795490 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:22:29.798166 (+ 2676us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:22:29.798177 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:22:29.798185 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:22:29.798234 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:22:29.802111 (+ 3877us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:22:29.802720 (+ 609us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:22:29.802728 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:22:29.802734 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:22:29.802930 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:22:29.803466 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:22:29.803474 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:22:29.803480 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:22:29.803872 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:22:29.805442 (+ 1570us) server_negotiation.cc:1035] Waiting for connection context
0929 20:22:29.805565 (+ 123us) server_negotiation.cc:299] Negotiation successful
0929 20:22:29.805792 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260929 20:22:29.808084 18964 heartbeater.cc:347] Connected to a master server at 127.17.53.126:43983
I20260929 20:22:29.808405 18964 heartbeater.cc:464] Registering TS with master...
I20260929 20:22:29.808813 17621 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu as pid 18826
I20260929 20:22:29.809111 17621 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790713339161492-17621-0/minicluster-data/ts-0/wal/instance
I20260929 20:22:29.809319 18964 heartbeater.cc:511] Master 127.17.53.126:43983 requested a full tablet report, sending...
I20260929 20:22:29.810108 17621 external_mini_cluster.cc:1428] Setting key 54ea084cb2bf594c3c136f6ca17bb143
I20260929 20:22:29.812099 18754 ts_manager.cc:194] Registered new tserver with Master: 744ee3020a694d7fad3c0ee1c7e50a26 (127.17.53.65:37593)
I20260929 20:22:29.813742 18754 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.53.65@KRBTEST.COM'} at 127.17.53.65:45133
WARNING: no policy specified for kudu/127.17.53.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.53.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.53.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.66.keytab.
Entry for principal kudu/127.17.53.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.66.keytab.
I20260929 20:22:29.850116 17621 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
/tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790713339161492-17621-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790713339161492-17621-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790713339161492-17621-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790713339161492-17621-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.53.66:0
--local_ip_for_outbound_sockets=127.17.53.66
--webserver_interface=127.17.53.66
--webserver_port=0
--tserver_master_addrs=127.17.53.126:43983
--keytab_file=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.66.keytab
--principal=kudu/127.17.53.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.53.84:34623
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790713339161492-17621-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790713339161492-17621-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790713339161492-17621-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:22:30.194113 18968 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:22:30.194443 18968 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:22:30.194617 18968 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:22:30.201869 18968 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:22:30.202025 18968 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:22:30.202194 18968 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 20:22:30.202342 18968 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.53.66
I20260929 20:22:30.212486 18968 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.53.84:34623
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790713339161492-17621-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790713339161492-17621-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.66.keytab
--principal=kudu/127.17.53.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.53.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790713339161492-17621-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.17.53.66
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.17.53.126:43983
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.53.66
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790713339161492-17621-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:22:30.213815 18968 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:22:30.215163 18968 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:22:30.228410 18976 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:22:30.228494 18973 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:22:30.228457 18974 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:22:30.229589 18968 server_base.cc:1034] running on GCE node
I20260929 20:22:30.230229 18968 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:22:30.231158 18968 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:22:30.232380 18968 hybrid_clock.cc:648] HybridClock initialized: now 1790713350232316 us; error 34 us; skew 500 ppm
I20260929 20:22:30.232700 18968 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:22:30 dist-test-slave-zbnc krb5kdc[18708](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713350, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.53.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:22:30.235867 18968 init.cc:377] Logged in from keytab as kudu/127.17.53.66@KRBTEST.COM (short username kudu)
I20260929 20:22:30.237347 18968 webserver.cc:460] Webserver started at http://127.17.53.66:33073/ using document root <none> and password file <none>
I20260929 20:22:30.238037 18968 fs_manager.cc:359] Metadata directory not provided
I20260929 20:22:30.238121 18968 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:22:30.238391 18968 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:22:30.241267 18968 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790713339161492-17621-0/minicluster-data/ts-1/data/instance:
uuid: "97175f85e6784f6bb1c4f433104a3802"
format_stamp: "Formatted at 2026-09-29 20:22:30 on dist-test-slave-zbnc"
server_key: "ba0f0af67a63800e0fc82d41845e19ca"
server_key_iv: "0ddfa814745506237063e644ac610e72"
server_key_version: "encryptionkey@0"
I20260929 20:22:30.242007 18968 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790713339161492-17621-0/minicluster-data/ts-1/wal/instance:
uuid: "97175f85e6784f6bb1c4f433104a3802"
format_stamp: "Formatted at 2026-09-29 20:22:30 on dist-test-slave-zbnc"
server_key: "ba0f0af67a63800e0fc82d41845e19ca"
server_key_iv: "0ddfa814745506237063e644ac610e72"
server_key_version: "encryptionkey@0"
I20260929 20:22:30.247056 18968 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260929 20:22:30.250324 18983 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:22:30.251814 18968 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 20:22:30.251984 18968 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790713339161492-17621-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790713339161492-17621-0/minicluster-data/ts-1/wal
uuid: "97175f85e6784f6bb1c4f433104a3802"
format_stamp: "Formatted at 2026-09-29 20:22:30 on dist-test-slave-zbnc"
server_key: "ba0f0af67a63800e0fc82d41845e19ca"
server_key_iv: "0ddfa814745506237063e644ac610e72"
server_key_version: "encryptionkey@0"
I20260929 20:22:30.252203 18968 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790713339161492-17621-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790713339161492-17621-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790713339161492-17621-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:22:30.277850 18968 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:22:30.282012 18968 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:22:30.282311 18968 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:22:30.285408 18968 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:22:30.285511 18968 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:22:30.285727 18968 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:22:30.285810 18968 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 20:22:30 dist-test-slave-zbnc krb5kdc[18708](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713350, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.53.66@KRBTEST.COM for kudu/127.17.53.126@KRBTEST.COM
I20260929 20:22:30.309244 18850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:30.285856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.53.66:50281 (local address 127.17.53.126:43983)
0929 20:22:30.286035 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0929 20:22:30.286043 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 20:22:30.287685 (+ 1642us) server_negotiation.cc:407] Connection header received
0929 20:22:30.289895 (+ 2210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:22:30.289901 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:22:30.289998 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:22:30.290110 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:22:30.292644 (+ 2534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:30.293740 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:30.295946 (+ 2206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:30.296392 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:30.296530 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:22:30.301658 (+ 5128us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:22:30.301663 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:22:30.301666 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:22:30.301700 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:22:30.304906 (+ 3206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:22:30.305960 (+ 1054us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:22:30.305966 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:22:30.305970 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:22:30.306097 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:22:30.306762 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:22:30.306767 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:22:30.306769 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:22:30.307186 (+ 417us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:22:30.307349 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0929 20:22:30.308420 (+ 1071us) server_negotiation.cc:299] Negotiation successful
0929 20:22:30.308724 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260929 20:22:30.310094 18993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:30.286605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:43983 (local address 127.17.53.66:50281)
0929 20:22:30.287530 (+ 925us) negotiation.cc:107] Waiting for socket to connect
0929 20:22:30.287575 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0929 20:22:30.289602 (+ 2027us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:22:30.290424 (+ 822us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:22:30.290439 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:22:30.291357 (+ 918us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:22:30.292382 (+ 1025us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:22:30.292419 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:22:30.293902 (+ 1483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:22:30.293907 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:22:30.295792 (+ 1885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:22:30.295806 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:22:30.296639 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:22:30.296647 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:22:30.296962 (+ 315us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:22:30.297789 (+ 827us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:22:30.297829 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:22:30.301390 (+ 3561us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:22:30.305069 (+ 3679us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:22:30.305081 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:22:30.305100 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:22:30.305824 (+ 724us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:22:30.306221 (+ 397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:22:30.306227 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:22:30.306230 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:22:30.306594 (+ 364us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:22:30.307407 (+ 813us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:22:30.307413 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:22:30.307947 (+ 534us) client_negotiation.cc:769] Sending connection context
0929 20:22:30.308291 (+ 344us) client_negotiation.cc:240] Negotiation successful
0929 20:22:30.308814 (+ 523us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":584,"thread_start_us":379,"threads_started":1}
I20260929 20:22:30.330533 18850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:30.316319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.53.66:50181 (local address 127.17.53.126:43983)
0929 20:22:30.316499 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0929 20:22:30.316504 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:22:30.316868 (+ 364us) server_negotiation.cc:407] Connection header received
0929 20:22:30.317070 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:22:30.317074 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:22:30.317133 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:22:30.317213 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:22:30.318734 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:30.319710 (+ 976us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:30.321418 (+ 1708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:30.321831 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:30.321978 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:22:30.324188 (+ 2210us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:22:30.324196 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:22:30.324202 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:22:30.324245 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:22:30.327903 (+ 3658us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:22:30.328579 (+ 676us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:22:30.328587 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:22:30.328590 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:22:30.328752 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:22:30.329105 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:22:30.329111 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:22:30.329115 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:22:30.329434 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:22:30.329590 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0929 20:22:30.329982 (+ 392us) server_negotiation.cc:299] Negotiation successful
0929 20:22:30.330192 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 20:22:30.330740 19054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:30.316051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:43983 (local address 127.17.53.66:50181)
0929 20:22:30.316748 (+ 697us) negotiation.cc:107] Waiting for socket to connect
0929 20:22:30.316772 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0929 20:22:30.316946 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:22:30.317283 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:22:30.317292 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:22:30.318098 (+ 806us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:22:30.318577 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:22:30.318593 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:22:30.319879 (+ 1286us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:22:30.319884 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:22:30.321230 (+ 1346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:22:30.321244 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:22:30.322019 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:22:30.322025 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:22:30.322191 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:22:30.322937 (+ 746us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:22:30.322957 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:22:30.323982 (+ 1025us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:22:30.328041 (+ 4059us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:22:30.328045 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:22:30.328049 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:22:30.328464 (+ 415us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:22:30.328869 (+ 405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:22:30.328874 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:22:30.328878 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:22:30.329001 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:22:30.329555 (+ 554us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:22:30.329560 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:22:30.329738 (+ 178us) client_negotiation.cc:769] Sending connection context
0929 20:22:30.329892 (+ 154us) client_negotiation.cc:240] Negotiation successful
0929 20:22:30.330291 (+ 399us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":497,"thread_start_us":338,"threads_started":1}
I20260929 20:22:30.341338 18968 rpc_server.cc:307] RPC server started. Bound to: 127.17.53.66:46249
I20260929 20:22:30.341385 19104 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.53.66:46249 every 8 connection(s)
I20260929 20:22:30.342562 18968 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790713339161492-17621-0/minicluster-data/ts-1/data/info.pb
I20260929 20:22:30.344619 19105 heartbeater.cc:347] Connected to a master server at 127.17.53.126:43983
I20260929 20:22:30.344857 19105 heartbeater.cc:464] Registering TS with master...
I20260929 20:22:30.345463 19105 heartbeater.cc:511] Master 127.17.53.126:43983 requested a full tablet report, sending...
I20260929 20:22:30.346684 18754 ts_manager.cc:194] Registered new tserver with Master: 97175f85e6784f6bb1c4f433104a3802 (127.17.53.66:46249)
I20260929 20:22:30.347569 18754 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.53.66@KRBTEST.COM'} at 127.17.53.66:50281
I20260929 20:22:30.349224 17621 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu as pid 18968
I20260929 20:22:30.349433 17621 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790713339161492-17621-0/minicluster-data/ts-1/wal/instance
I20260929 20:22:30.349877 17621 external_mini_cluster.cc:1428] Setting key 902520dc5049aa2425e2076bae7433e0
WARNING: no policy specified for kudu/127.17.53.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.53.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.53.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.67.keytab.
Entry for principal kudu/127.17.53.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.67.keytab.
I20260929 20:22:30.390080 17621 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
/tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790713339161492-17621-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790713339161492-17621-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790713339161492-17621-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790713339161492-17621-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.53.67:0
--local_ip_for_outbound_sockets=127.17.53.67
--webserver_interface=127.17.53.67
--webserver_port=0
--tserver_master_addrs=127.17.53.126:43983
--keytab_file=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.67.keytab
--principal=kudu/127.17.53.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.53.84:34623
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790713339161492-17621-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790713339161492-17621-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790713339161492-17621-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:22:30.715751 19109 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:22:30.716074 19109 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:22:30.716257 19109 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:22:30.723646 19109 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:22:30.723820 19109 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:22:30.723971 19109 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 20:22:30.724105 19109 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.53.67
I20260929 20:22:30.734210 19109 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.53.84:34623
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790713339161492-17621-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790713339161492-17621-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.67.keytab
--principal=kudu/127.17.53.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.53.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790713339161492-17621-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.17.53.67
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.17.53.126:43983
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.53.67
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790713339161492-17621-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:22:30.735612 19109 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:22:30.737006 19109 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:22:30.750532 19115 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:22:30.750659 19114 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:22:30.750663 19117 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:22:30.751945 19109 server_base.cc:1034] running on GCE node
I20260929 20:22:30.752676 19109 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:22:30.753700 19109 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:22:30.754980 19109 hybrid_clock.cc:648] HybridClock initialized: now 1790713350754885 us; error 64 us; skew 500 ppm
I20260929 20:22:30.755355 19109 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:22:30 dist-test-slave-zbnc krb5kdc[18708](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713350, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.53.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:22:30.758659 19109 init.cc:377] Logged in from keytab as kudu/127.17.53.67@KRBTEST.COM (short username kudu)
I20260929 20:22:30.760229 19109 webserver.cc:460] Webserver started at http://127.17.53.67:40245/ using document root <none> and password file <none>
I20260929 20:22:30.760972 19109 fs_manager.cc:359] Metadata directory not provided
I20260929 20:22:30.761103 19109 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:22:30.761440 19109 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:22:30.764535 19109 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790713339161492-17621-0/minicluster-data/ts-2/data/instance:
uuid: "fe48f25970b145629c3e36b146488cb0"
format_stamp: "Formatted at 2026-09-29 20:22:30 on dist-test-slave-zbnc"
server_key: "3bfe42d358efde3c2f503a88f84d708b"
server_key_iv: "92d47559c88b33aa92a413a33661ffbd"
server_key_version: "encryptionkey@0"
I20260929 20:22:30.765363 19109 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790713339161492-17621-0/minicluster-data/ts-2/wal/instance:
uuid: "fe48f25970b145629c3e36b146488cb0"
format_stamp: "Formatted at 2026-09-29 20:22:30 on dist-test-slave-zbnc"
server_key: "3bfe42d358efde3c2f503a88f84d708b"
server_key_iv: "92d47559c88b33aa92a413a33661ffbd"
server_key_version: "encryptionkey@0"
I20260929 20:22:30.770812 19109 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260929 20:22:30.774493 19124 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:22:30.776475 19109 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 20:22:30.776716 19109 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790713339161492-17621-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790713339161492-17621-0/minicluster-data/ts-2/wal
uuid: "fe48f25970b145629c3e36b146488cb0"
format_stamp: "Formatted at 2026-09-29 20:22:30 on dist-test-slave-zbnc"
server_key: "3bfe42d358efde3c2f503a88f84d708b"
server_key_iv: "92d47559c88b33aa92a413a33661ffbd"
server_key_version: "encryptionkey@0"
I20260929 20:22:30.776995 19109 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790713339161492-17621-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790713339161492-17621-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790713339161492-17621-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:22:30.804536 19109 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:22:30.809957 19109 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:22:30.810274 19109 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:22:30.813336 19109 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:22:30.813423 19109 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:22:30.813580 19109 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:22:30.813678 19109 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:22:30.816481 18964 heartbeater.cc:503] Master 127.17.53.126:43983 was elected leader, sending a full tablet report...
Sep 29 20:22:30 dist-test-slave-zbnc krb5kdc[18708](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713350, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.53.67@KRBTEST.COM for kudu/127.17.53.126@KRBTEST.COM
I20260929 20:22:30.837967 18850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:30.813960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.53.67:36505 (local address 127.17.53.126:43983)
0929 20:22:30.814158 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0929 20:22:30.814166 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 20:22:30.815893 (+ 1727us) server_negotiation.cc:407] Connection header received
0929 20:22:30.818598 (+ 2705us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:22:30.818607 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:22:30.818710 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:22:30.818808 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:22:30.821549 (+ 2741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:30.822668 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:30.825252 (+ 2584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:30.826014 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:30.826200 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:22:30.831768 (+ 5568us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:22:30.831773 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:22:30.831776 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:22:30.831809 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:22:30.834520 (+ 2711us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:22:30.835385 (+ 865us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:22:30.835395 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:22:30.835398 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:22:30.835531 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:22:30.836019 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:22:30.836024 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:22:30.836027 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:22:30.836308 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:22:30.836466 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0929 20:22:30.837533 (+ 1067us) server_negotiation.cc:299] Negotiation successful
0929 20:22:30.837713 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260929 20:22:30.841331 19133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:30.814666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:43983 (local address 127.17.53.67:36505)
0929 20:22:30.815673 (+ 1007us) negotiation.cc:107] Waiting for socket to connect
0929 20:22:30.815731 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0929 20:22:30.818237 (+ 2506us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:22:30.819430 (+ 1193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:22:30.819444 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:22:30.820333 (+ 889us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:22:30.821348 (+ 1015us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:22:30.821383 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:22:30.822884 (+ 1501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:22:30.822893 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:22:30.825090 (+ 2197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:22:30.825101 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:22:30.826352 (+ 1251us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:22:30.826364 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:22:30.826793 (+ 429us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:22:30.827682 (+ 889us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:22:30.827724 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:22:30.831526 (+ 3802us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:22:30.834675 (+ 3149us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:22:30.834684 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:22:30.834701 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:22:30.835253 (+ 552us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:22:30.835671 (+ 418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:22:30.835676 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:22:30.835680 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:22:30.835889 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:22:30.836447 (+ 558us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:22:30.836456 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:22:30.837034 (+ 578us) client_negotiation.cc:769] Sending connection context
0929 20:22:30.837414 (+ 380us) client_negotiation.cc:240] Negotiation successful
0929 20:22:30.838712 (+ 1298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":707,"thread_start_us":491,"threads_started":1}
I20260929 20:22:30.862649 19109 rpc_server.cc:307] RPC server started. Bound to: 127.17.53.67:34979
I20260929 20:22:30.862691 19244 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.53.67:34979 every 8 connection(s)
I20260929 20:22:30.863971 19109 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790713339161492-17621-0/minicluster-data/ts-2/data/info.pb
I20260929 20:22:30.866932 19245 heartbeater.cc:347] Connected to a master server at 127.17.53.126:43983
I20260929 20:22:30.867292 19245 heartbeater.cc:464] Registering TS with master...
I20260929 20:22:30.867460 17621 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu as pid 19109
I20260929 20:22:30.867652 17621 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790713339161492-17621-0/minicluster-data/ts-2/wal/instance
I20260929 20:22:30.867930 18850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:30.850787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.53.67:43423 (local address 127.17.53.126:43983)
0929 20:22:30.851036 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0929 20:22:30.851045 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 20:22:30.851810 (+ 765us) server_negotiation.cc:407] Connection header received
0929 20:22:30.852046 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:22:30.852052 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:22:30.852132 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:22:30.852235 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:22:30.853896 (+ 1661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:30.855133 (+ 1237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:30.858083 (+ 2950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:30.858631 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:30.858801 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:22:30.861479 (+ 2678us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:22:30.861486 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:22:30.861490 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:22:30.861528 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:22:30.864830 (+ 3302us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:22:30.865664 (+ 834us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:22:30.865670 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:22:30.865674 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:22:30.865841 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:22:30.866295 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:22:30.866301 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:22:30.866305 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:22:30.866673 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:22:30.866843 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0929 20:22:30.867317 (+ 474us) server_negotiation.cc:299] Negotiation successful
0929 20:22:30.867563 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260929 20:22:30.868037 19220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:30.851057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:43983 (local address 127.17.53.67:43423)
0929 20:22:30.851675 (+ 618us) negotiation.cc:107] Waiting for socket to connect
0929 20:22:30.851694 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 20:22:30.851898 (+ 204us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:22:30.852259 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:22:30.852264 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:22:30.853052 (+ 788us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:22:30.853662 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:22:30.853683 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:22:30.855329 (+ 1646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:22:30.855338 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:22:30.857901 (+ 2563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:22:30.857924 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:22:30.858925 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:22:30.858932 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:22:30.859196 (+ 264us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:22:30.860066 (+ 870us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:22:30.860104 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:22:30.861244 (+ 1140us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:22:30.864998 (+ 3754us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:22:30.865003 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:22:30.865006 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:22:30.865545 (+ 539us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:22:30.865980 (+ 435us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:22:30.865984 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:22:30.865986 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:22:30.866163 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:22:30.866803 (+ 640us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:22:30.866809 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:22:30.867030 (+ 221us) client_negotiation.cc:769] Sending connection context
0929 20:22:30.867206 (+ 176us) client_negotiation.cc:240] Negotiation successful
0929 20:22:30.867559 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":484,"thread_start_us":352,"threads_started":1}
I20260929 20:22:30.868209 17621 external_mini_cluster.cc:1428] Setting key 11d468f972c5f416057a10a2d2675aa1
I20260929 20:22:30.868402 19245 heartbeater.cc:511] Master 127.17.53.126:43983 requested a full tablet report, sending...
I20260929 20:22:30.870141 18754 ts_manager.cc:194] Registered new tserver with Master: fe48f25970b145629c3e36b146488cb0 (127.17.53.67:34979)
I20260929 20:22:30.871217 18754 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.53.67@KRBTEST.COM'} at 127.17.53.67:36505
I20260929 20:22:30.871759 17621 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 29 20:22:30 dist-test-slave-zbnc krb5kdc[18708](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713350, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 29 20:22:30 dist-test-slave-zbnc krb5kdc[18708](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713350, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.17.53.126@KRBTEST.COM
I20260929 20:22:30.918694 18850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:30.904579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49096 (local address 127.17.53.126:43983)
0929 20:22:30.904812 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0929 20:22:30.904820 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 20:22:30.905045 (+ 225us) server_negotiation.cc:407] Connection header received
0929 20:22:30.905291 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:22:30.905296 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:22:30.905361 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:22:30.905475 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:22:30.906940 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:30.907692 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:30.909011 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:30.909443 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:30.909585 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:22:30.913603 (+ 4018us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:22:30.913608 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:22:30.913611 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:22:30.913647 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:22:30.916079 (+ 2432us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:22:30.916711 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:22:30.916717 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:22:30.916720 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:22:30.916843 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:22:30.917239 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:22:30.917244 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:22:30.917248 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:22:30.917529 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:22:30.917783 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0929 20:22:30.918113 (+ 330us) server_negotiation.cc:299] Negotiation successful
0929 20:22:30.918353 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260929 20:22:30.924849 18754 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49096:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 20:22:30.934702 18754 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 20:22:30.968377 19260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:30.950882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37860 (local address 127.17.53.65:37593)
0929 20:22:30.951784 (+ 902us) server_negotiation.cc:206] Beginning negotiation
0929 20:22:30.951793 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 20:22:30.955570 (+ 3777us) server_negotiation.cc:407] Connection header received
0929 20:22:30.955780 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:22:30.955793 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:22:30.956137 (+ 344us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:22:30.956990 (+ 853us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:22:30.959085 (+ 2095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:30.962804 (+ 3719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:30.965337 (+ 2533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:30.966562 (+ 1225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:30.966710 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:22:30.966974 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0929 20:22:30.967710 (+ 736us) server_negotiation.cc:299] Negotiation successful
0929 20:22:30.967975 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":730,"thread_start_us":586,"threads_started":1}
I20260929 20:22:30.968434 19259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:30.950583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.65:37593 (local address 127.0.0.1:37860)
0929 20:22:30.955233 (+ 4650us) negotiation.cc:107] Waiting for socket to connect
0929 20:22:30.955257 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0929 20:22:30.955410 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:22:30.958076 (+ 2666us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:22:30.958088 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:22:30.958130 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:22:30.958705 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:22:30.958724 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:22:30.963127 (+ 4403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:22:30.963140 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:22:30.965143 (+ 2003us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:22:30.965155 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:22:30.967119 (+ 1964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:22:30.967125 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:22:30.967384 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:22:30.967454 (+ 70us) client_negotiation.cc:769] Sending connection context
0929 20:22:30.967635 (+ 181us) client_negotiation.cc:240] Negotiation successful
0929 20:22:30.967925 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4482,"mutex_wait_us":49,"thread_start_us":1446,"threads_started":1}
I20260929 20:22:30.970464 19258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:30.949874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.67:34979 (local address 127.0.0.1:49554)
0929 20:22:30.950923 (+ 1049us) negotiation.cc:107] Waiting for socket to connect
0929 20:22:30.950961 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0929 20:22:30.951152 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:22:30.962806 (+ 11654us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:22:30.962811 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:22:30.962836 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:22:30.963421 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:22:30.963439 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:22:30.965594 (+ 2155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:22:30.965599 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:22:30.967726 (+ 2127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:22:30.967742 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:22:30.969633 (+ 1891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:22:30.969639 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:22:30.969823 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:22:30.969857 (+ 34us) client_negotiation.cc:769] Sending connection context
0929 20:22:30.969936 (+ 79us) client_negotiation.cc:240] Negotiation successful
0929 20:22:30.970068 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":548,"thread_start_us":2881,"threads_started":1}
I20260929 20:22:30.971621 19262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:30.951252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49554 (local address 127.17.53.67:34979)
0929 20:22:30.962086 (+ 10834us) server_negotiation.cc:206] Beginning negotiation
0929 20:22:30.962097 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 20:22:30.962144 (+ 47us) server_negotiation.cc:407] Connection header received
0929 20:22:30.962255 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:22:30.962264 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:22:30.962577 (+ 313us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:22:30.962806 (+ 229us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:22:30.963702 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:30.965455 (+ 1753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:30.967943 (+ 2488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:30.969405 (+ 1462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:30.970685 (+ 1280us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:22:30.970963 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0929 20:22:30.971043 (+ 80us) server_negotiation.cc:299] Negotiation successful
0929 20:22:30.971222 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10624,"thread_start_us":11849,"threads_started":1}
I20260929 20:22:30.973852 19263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:30.951845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48808 (local address 127.17.53.66:46249)
0929 20:22:30.952348 (+ 503us) server_negotiation.cc:206] Beginning negotiation
0929 20:22:30.952358 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 20:22:30.964107 (+ 11749us) server_negotiation.cc:407] Connection header received
0929 20:22:30.964228 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:22:30.964241 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:22:30.964510 (+ 269us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:22:30.964700 (+ 190us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:22:30.965569 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:30.967084 (+ 1515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:30.969696 (+ 2612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:30.971390 (+ 1694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:30.971525 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:22:30.973323 (+ 1798us) server_negotiation.cc:1035] Waiting for connection context
0929 20:22:30.973407 (+ 84us) server_negotiation.cc:299] Negotiation successful
0929 20:22:30.973568 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":357,"thread_start_us":751,"threads_started":1}
I20260929 20:22:30.973390 18893 tablet_service.cc:1467] Processing CreateTablet for tablet 640fdae70a5b460bb321a69b68b1aa35 (DEFAULT_TABLE table=test-table [id=ff77cc6bd5014ebd9b88fe6588450366]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:22:30.974486 19261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:30.950935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.66:46249 (local address 127.0.0.1:48808)
0929 20:22:30.963818 (+ 12883us) negotiation.cc:107] Waiting for socket to connect
0929 20:22:30.963843 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0929 20:22:30.963973 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:22:30.964937 (+ 964us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:22:30.964942 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:22:30.964965 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:22:30.965347 (+ 382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:22:30.965357 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:22:30.967246 (+ 1889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:22:30.967255 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:22:30.969534 (+ 2279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:22:30.969548 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:22:30.971940 (+ 2392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:22:30.971946 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:22:30.972115 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:22:30.972144 (+ 29us) client_negotiation.cc:769] Sending connection context
0929 20:22:30.972224 (+ 80us) client_negotiation.cc:240] Negotiation successful
0929 20:22:30.974190 (+ 1966us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12738,"thread_start_us":12563,"threads_started":1}
I20260929 20:22:30.974997 18893 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 640fdae70a5b460bb321a69b68b1aa35. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:22:30.975369 19174 tablet_service.cc:1467] Processing CreateTablet for tablet 640fdae70a5b460bb321a69b68b1aa35 (DEFAULT_TABLE table=test-table [id=ff77cc6bd5014ebd9b88fe6588450366]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:22:30.976890 19174 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 640fdae70a5b460bb321a69b68b1aa35. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:22:30.986261 19034 tablet_service.cc:1467] Processing CreateTablet for tablet 640fdae70a5b460bb321a69b68b1aa35 (DEFAULT_TABLE table=test-table [id=ff77cc6bd5014ebd9b88fe6588450366]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:22:30.988153 19034 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 640fdae70a5b460bb321a69b68b1aa35. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:22:30.989569 19264 tablet_bootstrap.cc:492] T 640fdae70a5b460bb321a69b68b1aa35 P 744ee3020a694d7fad3c0ee1c7e50a26: Bootstrap starting.
I20260929 20:22:30.991585 19265 tablet_bootstrap.cc:492] T 640fdae70a5b460bb321a69b68b1aa35 P fe48f25970b145629c3e36b146488cb0: Bootstrap starting.
I20260929 20:22:30.993211 19264 tablet_bootstrap.cc:654] T 640fdae70a5b460bb321a69b68b1aa35 P 744ee3020a694d7fad3c0ee1c7e50a26: Neither blocks nor log segments found. Creating new log.
I20260929 20:22:30.994590 19264 log.cc:824] T 640fdae70a5b460bb321a69b68b1aa35 P 744ee3020a694d7fad3c0ee1c7e50a26: Log is configured to *not* fsync() on all Append() calls
I20260929 20:22:30.995055 19265 tablet_bootstrap.cc:654] T 640fdae70a5b460bb321a69b68b1aa35 P fe48f25970b145629c3e36b146488cb0: Neither blocks nor log segments found. Creating new log.
I20260929 20:22:30.996232 19265 log.cc:824] T 640fdae70a5b460bb321a69b68b1aa35 P fe48f25970b145629c3e36b146488cb0: Log is configured to *not* fsync() on all Append() calls
I20260929 20:22:30.997694 19264 tablet_bootstrap.cc:492] T 640fdae70a5b460bb321a69b68b1aa35 P 744ee3020a694d7fad3c0ee1c7e50a26: No bootstrap required, opened a new log
I20260929 20:22:30.997969 19264 ts_tablet_manager.cc:1397] T 640fdae70a5b460bb321a69b68b1aa35 P 744ee3020a694d7fad3c0ee1c7e50a26: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260929 20:22:30.999300 19265 tablet_bootstrap.cc:492] T 640fdae70a5b460bb321a69b68b1aa35 P fe48f25970b145629c3e36b146488cb0: No bootstrap required, opened a new log
I20260929 20:22:30.999553 19265 ts_tablet_manager.cc:1397] T 640fdae70a5b460bb321a69b68b1aa35 P fe48f25970b145629c3e36b146488cb0: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.006s
I20260929 20:22:31.003441 19268 tablet_bootstrap.cc:492] T 640fdae70a5b460bb321a69b68b1aa35 P 97175f85e6784f6bb1c4f433104a3802: Bootstrap starting.
I20260929 20:22:31.007110 19268 tablet_bootstrap.cc:654] T 640fdae70a5b460bb321a69b68b1aa35 P 97175f85e6784f6bb1c4f433104a3802: Neither blocks nor log segments found. Creating new log.
I20260929 20:22:31.007984 19265 raft_consensus.cc:348] T 640fdae70a5b460bb321a69b68b1aa35 P fe48f25970b145629c3e36b146488cb0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe48f25970b145629c3e36b146488cb0" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 34979 } } peers { permanent_uuid: "744ee3020a694d7fad3c0ee1c7e50a26" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 37593 } } peers { permanent_uuid: "97175f85e6784f6bb1c4f433104a3802" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 46249 } }
I20260929 20:22:31.008394 19265 raft_consensus.cc:374] T 640fdae70a5b460bb321a69b68b1aa35 P fe48f25970b145629c3e36b146488cb0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:22:31.008486 19265 raft_consensus.cc:729] T 640fdae70a5b460bb321a69b68b1aa35 P fe48f25970b145629c3e36b146488cb0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe48f25970b145629c3e36b146488cb0, State: Initialized, Role: FOLLOWER
I20260929 20:22:31.008487 19268 log.cc:824] T 640fdae70a5b460bb321a69b68b1aa35 P 97175f85e6784f6bb1c4f433104a3802: Log is configured to *not* fsync() on all Append() calls
I20260929 20:22:31.008584 19264 raft_consensus.cc:348] T 640fdae70a5b460bb321a69b68b1aa35 P 744ee3020a694d7fad3c0ee1c7e50a26 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe48f25970b145629c3e36b146488cb0" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 34979 } } peers { permanent_uuid: "744ee3020a694d7fad3c0ee1c7e50a26" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 37593 } } peers { permanent_uuid: "97175f85e6784f6bb1c4f433104a3802" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 46249 } }
I20260929 20:22:31.009037 19264 raft_consensus.cc:374] T 640fdae70a5b460bb321a69b68b1aa35 P 744ee3020a694d7fad3c0ee1c7e50a26 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:22:31.009173 19264 raft_consensus.cc:729] T 640fdae70a5b460bb321a69b68b1aa35 P 744ee3020a694d7fad3c0ee1c7e50a26 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 744ee3020a694d7fad3c0ee1c7e50a26, State: Initialized, Role: FOLLOWER
I20260929 20:22:31.009168 19265 consensus_queue.cc:260] T 640fdae70a5b460bb321a69b68b1aa35 P fe48f25970b145629c3e36b146488cb0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe48f25970b145629c3e36b146488cb0" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 34979 } } peers { permanent_uuid: "744ee3020a694d7fad3c0ee1c7e50a26" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 37593 } } peers { permanent_uuid: "97175f85e6784f6bb1c4f433104a3802" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 46249 } }
I20260929 20:22:31.009795 19264 consensus_queue.cc:260] T 640fdae70a5b460bb321a69b68b1aa35 P 744ee3020a694d7fad3c0ee1c7e50a26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe48f25970b145629c3e36b146488cb0" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 34979 } } peers { permanent_uuid: "744ee3020a694d7fad3c0ee1c7e50a26" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 37593 } } peers { permanent_uuid: "97175f85e6784f6bb1c4f433104a3802" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 46249 } }
I20260929 20:22:31.010385 19245 heartbeater.cc:503] Master 127.17.53.126:43983 was elected leader, sending a full tablet report...
I20260929 20:22:31.011047 19265 ts_tablet_manager.cc:1428] T 640fdae70a5b460bb321a69b68b1aa35 P fe48f25970b145629c3e36b146488cb0: Time spent starting tablet: real 0.011s user 0.012s sys 0.000s
I20260929 20:22:31.012193 19268 tablet_bootstrap.cc:492] T 640fdae70a5b460bb321a69b68b1aa35 P 97175f85e6784f6bb1c4f433104a3802: No bootstrap required, opened a new log
I20260929 20:22:31.012475 19268 ts_tablet_manager.cc:1397] T 640fdae70a5b460bb321a69b68b1aa35 P 97175f85e6784f6bb1c4f433104a3802: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.006s
I20260929 20:22:31.013530 19264 ts_tablet_manager.cc:1428] T 640fdae70a5b460bb321a69b68b1aa35 P 744ee3020a694d7fad3c0ee1c7e50a26: Time spent starting tablet: real 0.015s user 0.009s sys 0.008s
I20260929 20:22:31.020681 19268 raft_consensus.cc:348] T 640fdae70a5b460bb321a69b68b1aa35 P 97175f85e6784f6bb1c4f433104a3802 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe48f25970b145629c3e36b146488cb0" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 34979 } } peers { permanent_uuid: "744ee3020a694d7fad3c0ee1c7e50a26" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 37593 } } peers { permanent_uuid: "97175f85e6784f6bb1c4f433104a3802" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 46249 } }
I20260929 20:22:31.021016 19268 raft_consensus.cc:374] T 640fdae70a5b460bb321a69b68b1aa35 P 97175f85e6784f6bb1c4f433104a3802 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:22:31.021090 19268 raft_consensus.cc:729] T 640fdae70a5b460bb321a69b68b1aa35 P 97175f85e6784f6bb1c4f433104a3802 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97175f85e6784f6bb1c4f433104a3802, State: Initialized, Role: FOLLOWER
I20260929 20:22:31.021544 19268 consensus_queue.cc:260] T 640fdae70a5b460bb321a69b68b1aa35 P 97175f85e6784f6bb1c4f433104a3802 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe48f25970b145629c3e36b146488cb0" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 34979 } } peers { permanent_uuid: "744ee3020a694d7fad3c0ee1c7e50a26" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 37593 } } peers { permanent_uuid: "97175f85e6784f6bb1c4f433104a3802" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 46249 } }
I20260929 20:22:31.022423 19105 heartbeater.cc:503] Master 127.17.53.126:43983 was elected leader, sending a full tablet report...
I20260929 20:22:31.022838 19268 ts_tablet_manager.cc:1428] T 640fdae70a5b460bb321a69b68b1aa35 P 97175f85e6784f6bb1c4f433104a3802: Time spent starting tablet: real 0.010s user 0.010s sys 0.001s
I20260929 20:22:31.035841 19272 raft_consensus.cc:482] T 640fdae70a5b460bb321a69b68b1aa35 P 97175f85e6784f6bb1c4f433104a3802 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 20:22:31.036159 19272 raft_consensus.cc:504] T 640fdae70a5b460bb321a69b68b1aa35 P 97175f85e6784f6bb1c4f433104a3802 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe48f25970b145629c3e36b146488cb0" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 34979 } } peers { permanent_uuid: "744ee3020a694d7fad3c0ee1c7e50a26" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 37593 } } peers { permanent_uuid: "97175f85e6784f6bb1c4f433104a3802" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 46249 } }
I20260929 20:22:31.037779 19272 leader_election.cc:288] T 640fdae70a5b460bb321a69b68b1aa35 P 97175f85e6784f6bb1c4f433104a3802 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fe48f25970b145629c3e36b146488cb0 (127.17.53.67:34979), 744ee3020a694d7fad3c0ee1c7e50a26 (127.17.53.65:37593)
I20260929 20:22:31.044673 19260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:31.038210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.53.66:38727 (local address 127.17.53.65:37593)
0929 20:22:31.038415 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0929 20:22:31.038421 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:22:31.039334 (+ 913us) server_negotiation.cc:407] Connection header received
0929 20:22:31.039518 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:22:31.039523 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:22:31.039590 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:22:31.039684 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:22:31.040354 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:31.041062 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:31.042883 (+ 1821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:31.043644 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:31.043824 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:22:31.043907 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0929 20:22:31.044310 (+ 403us) server_negotiation.cc:299] Negotiation successful
0929 20:22:31.044428 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 20:22:31.044703 19262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:31.038062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.53.66:46875 (local address 127.17.53.67:34979)
0929 20:22:31.038267 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0929 20:22:31.038273 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:22:31.039243 (+ 970us) server_negotiation.cc:407] Connection header received
0929 20:22:31.039456 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:22:31.039461 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:22:31.039518 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:22:31.039611 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:22:31.040241 (+ 630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:31.040959 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:31.042829 (+ 1870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:31.043716 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:31.043904 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:22:31.043975 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0929 20:22:31.044324 (+ 349us) server_negotiation.cc:299] Negotiation successful
0929 20:22:31.044443 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260929 20:22:31.044921 19274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:31.038267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.65:37593 (local address 127.17.53.66:38727)
0929 20:22:31.039239 (+ 972us) negotiation.cc:107] Waiting for socket to connect
0929 20:22:31.039253 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 20:22:31.039357 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:22:31.039696 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:22:31.039700 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:22:31.039727 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:22:31.040193 (+ 466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:22:31.040201 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:22:31.041199 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:22:31.041205 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:22:31.042696 (+ 1491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:22:31.042707 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:22:31.043882 (+ 1175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:22:31.043888 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:22:31.044109 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:22:31.044166 (+ 57us) client_negotiation.cc:769] Sending connection context
0929 20:22:31.044265 (+ 99us) client_negotiation.cc:240] Negotiation successful
0929 20:22:31.044389 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":881,"mutex_wait_us":40,"thread_start_us":449,"threads_started":1}
I20260929 20:22:31.044955 19273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:31.038225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.67:34979 (local address 127.17.53.66:46875)
0929 20:22:31.039139 (+ 914us) negotiation.cc:107] Waiting for socket to connect
0929 20:22:31.039154 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 20:22:31.039321 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:22:31.039624 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:22:31.039629 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:22:31.039650 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:22:31.040096 (+ 446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:22:31.040105 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:22:31.041084 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:22:31.041093 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:22:31.042674 (+ 1581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:22:31.042686 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:22:31.043929 (+ 1243us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:22:31.043935 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:22:31.044166 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:22:31.044188 (+ 22us) client_negotiation.cc:769] Sending connection context
0929 20:22:31.044289 (+ 101us) client_negotiation.cc:240] Negotiation successful
0929 20:22:31.044411 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":815,"mutex_wait_us":40,"spinlock_wait_cycles":22272,"thread_start_us":540,"threads_started":1}
I20260929 20:22:31.045876 19194 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "640fdae70a5b460bb321a69b68b1aa35" candidate_uuid: "97175f85e6784f6bb1c4f433104a3802" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe48f25970b145629c3e36b146488cb0" is_pre_election: true
I20260929 20:22:31.046072 18913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "640fdae70a5b460bb321a69b68b1aa35" candidate_uuid: "97175f85e6784f6bb1c4f433104a3802" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "744ee3020a694d7fad3c0ee1c7e50a26" is_pre_election: true
I20260929 20:22:31.046448 19194 raft_consensus.cc:2454] T 640fdae70a5b460bb321a69b68b1aa35 P fe48f25970b145629c3e36b146488cb0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97175f85e6784f6bb1c4f433104a3802 in term 0.
I20260929 20:22:31.046638 18913 raft_consensus.cc:2454] T 640fdae70a5b460bb321a69b68b1aa35 P 744ee3020a694d7fad3c0ee1c7e50a26 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97175f85e6784f6bb1c4f433104a3802 in term 0.
I20260929 20:22:31.047214 18986 leader_election.cc:302] T 640fdae70a5b460bb321a69b68b1aa35 P 97175f85e6784f6bb1c4f433104a3802 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 744ee3020a694d7fad3c0ee1c7e50a26, 97175f85e6784f6bb1c4f433104a3802, fe48f25970b145629c3e36b146488cb0; no voters:
I20260929 20:22:31.047683 19272 raft_consensus.cc:2781] T 640fdae70a5b460bb321a69b68b1aa35 P 97175f85e6784f6bb1c4f433104a3802 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 20:22:31.047845 19272 raft_consensus.cc:482] T 640fdae70a5b460bb321a69b68b1aa35 P 97175f85e6784f6bb1c4f433104a3802 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 20:22:31.047961 19272 raft_consensus.cc:3037] T 640fdae70a5b460bb321a69b68b1aa35 P 97175f85e6784f6bb1c4f433104a3802 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:22:31.050453 19272 raft_consensus.cc:504] T 640fdae70a5b460bb321a69b68b1aa35 P 97175f85e6784f6bb1c4f433104a3802 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe48f25970b145629c3e36b146488cb0" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 34979 } } peers { permanent_uuid: "744ee3020a694d7fad3c0ee1c7e50a26" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 37593 } } peers { permanent_uuid: "97175f85e6784f6bb1c4f433104a3802" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 46249 } }
I20260929 20:22:31.051244 19272 leader_election.cc:288] T 640fdae70a5b460bb321a69b68b1aa35 P 97175f85e6784f6bb1c4f433104a3802 [CANDIDATE]: Term 1 election: Requested vote from peers fe48f25970b145629c3e36b146488cb0 (127.17.53.67:34979), 744ee3020a694d7fad3c0ee1c7e50a26 (127.17.53.65:37593)
I20260929 20:22:31.051801 18913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "640fdae70a5b460bb321a69b68b1aa35" candidate_uuid: "97175f85e6784f6bb1c4f433104a3802" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "744ee3020a694d7fad3c0ee1c7e50a26"
I20260929 20:22:31.052013 18913 raft_consensus.cc:3037] T 640fdae70a5b460bb321a69b68b1aa35 P 744ee3020a694d7fad3c0ee1c7e50a26 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:22:31.052079 19194 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "640fdae70a5b460bb321a69b68b1aa35" candidate_uuid: "97175f85e6784f6bb1c4f433104a3802" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe48f25970b145629c3e36b146488cb0"
I20260929 20:22:31.052230 19194 raft_consensus.cc:3037] T 640fdae70a5b460bb321a69b68b1aa35 P fe48f25970b145629c3e36b146488cb0 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:22:31.054370 19194 raft_consensus.cc:2454] T 640fdae70a5b460bb321a69b68b1aa35 P fe48f25970b145629c3e36b146488cb0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97175f85e6784f6bb1c4f433104a3802 in term 1.
I20260929 20:22:31.054710 18986 leader_election.cc:302] T 640fdae70a5b460bb321a69b68b1aa35 P 97175f85e6784f6bb1c4f433104a3802 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 97175f85e6784f6bb1c4f433104a3802, fe48f25970b145629c3e36b146488cb0; no voters:
I20260929 20:22:31.055011 19272 raft_consensus.cc:2781] T 640fdae70a5b460bb321a69b68b1aa35 P 97175f85e6784f6bb1c4f433104a3802 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:22:31.055575 19272 raft_consensus.cc:686] T 640fdae70a5b460bb321a69b68b1aa35 P 97175f85e6784f6bb1c4f433104a3802 [term 1 LEADER]: Becoming Leader. State: Replica: 97175f85e6784f6bb1c4f433104a3802, State: Running, Role: LEADER
I20260929 20:22:31.055816 18913 raft_consensus.cc:2454] T 640fdae70a5b460bb321a69b68b1aa35 P 744ee3020a694d7fad3c0ee1c7e50a26 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97175f85e6784f6bb1c4f433104a3802 in term 1.
I20260929 20:22:31.056178 19272 consensus_queue.cc:237] T 640fdae70a5b460bb321a69b68b1aa35 P 97175f85e6784f6bb1c4f433104a3802 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe48f25970b145629c3e36b146488cb0" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 34979 } } peers { permanent_uuid: "744ee3020a694d7fad3c0ee1c7e50a26" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 37593 } } peers { permanent_uuid: "97175f85e6784f6bb1c4f433104a3802" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 46249 } }
I20260929 20:22:31.061522 18753 catalog_manager.cc:5495] T 640fdae70a5b460bb321a69b68b1aa35 P 97175f85e6784f6bb1c4f433104a3802 reported cstate change: term changed from 0 to 1, leader changed from <none> to 97175f85e6784f6bb1c4f433104a3802 (127.17.53.66). New cstate: current_term: 1 leader_uuid: "97175f85e6784f6bb1c4f433104a3802" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fe48f25970b145629c3e36b146488cb0" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 34979 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "744ee3020a694d7fad3c0ee1c7e50a26" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 37593 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "97175f85e6784f6bb1c4f433104a3802" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 46249 } health_report { overall_health: HEALTHY } } }
W20260929 20:22:31.064173 18965 tablet.cc:2369] T 640fdae70a5b460bb321a69b68b1aa35 P 744ee3020a694d7fad3c0ee1c7e50a26: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 20:22:31.093252 19263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:31.088163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48818 (local address 127.17.53.66:46249)
0929 20:22:31.088350 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0929 20:22:31.088354 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:22:31.088559 (+ 205us) server_negotiation.cc:407] Connection header received
0929 20:22:31.088786 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:22:31.088789 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:22:31.088871 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:22:31.088988 (+ 117us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 20:22:31.089290 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:31.089969 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:31.091333 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:31.091727 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:31.091824 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:22:31.092242 (+ 418us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 20:22:31.092754 (+ 512us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 20:22:31.092861 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0929 20:22:31.092973 (+ 112us) server_negotiation.cc:299] Negotiation successful
0929 20:22:31.093040 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
W20260929 20:22:31.094480 19106 tablet.cc:2369] T 640fdae70a5b460bb321a69b68b1aa35 P 97175f85e6784f6bb1c4f433104a3802: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 20:22:31.101438 19194 raft_consensus.cc:1261] T 640fdae70a5b460bb321a69b68b1aa35 P fe48f25970b145629c3e36b146488cb0 [term 1 FOLLOWER]: Refusing update from remote peer 97175f85e6784f6bb1c4f433104a3802: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:22:31.101497 18913 raft_consensus.cc:1261] T 640fdae70a5b460bb321a69b68b1aa35 P 744ee3020a694d7fad3c0ee1c7e50a26 [term 1 FOLLOWER]: Refusing update from remote peer 97175f85e6784f6bb1c4f433104a3802: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:22:31.102427 19275 consensus_queue.cc:1035] T 640fdae70a5b460bb321a69b68b1aa35 P 97175f85e6784f6bb1c4f433104a3802 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe48f25970b145629c3e36b146488cb0" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 34979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:22:31.102788 19272 consensus_queue.cc:1035] T 640fdae70a5b460bb321a69b68b1aa35 P 97175f85e6784f6bb1c4f433104a3802 [LEADER]: Connected to new peer: Peer: permanent_uuid: "744ee3020a694d7fad3c0ee1c7e50a26" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 37593 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:22:31.135679 18752 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49096:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 20:22:31.136052 18752 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49096:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 20:22:31.140590 18752 catalog_manager.cc:5782] T 00000000000000000000000000000000 P edc009a06883495fbef928dda0915f0d: Sending DeleteTablet for 3 replicas of tablet 640fdae70a5b460bb321a69b68b1aa35
I20260929 20:22:31.141803 19174 tablet_service.cc:1514] Processing DeleteTablet for tablet 640fdae70a5b460bb321a69b68b1aa35 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:22:31 UTC) from {username='kudu', principal='kudu/127.17.53.126@KRBTEST.COM'} at 127.0.0.1:49554
I20260929 20:22:31.141911 19034 tablet_service.cc:1514] Processing DeleteTablet for tablet 640fdae70a5b460bb321a69b68b1aa35 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:22:31 UTC) from {username='kudu', principal='kudu/127.17.53.126@KRBTEST.COM'} at 127.0.0.1:48808
I20260929 20:22:31.141911 18893 tablet_service.cc:1514] Processing DeleteTablet for tablet 640fdae70a5b460bb321a69b68b1aa35 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:22:31 UTC) from {username='kudu', principal='kudu/127.17.53.126@KRBTEST.COM'} at 127.0.0.1:37860
I20260929 20:22:31.144171 18752 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49096:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 20:22:31.145516 18752 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 20:22:31.148496 19288 tablet_replica.cc:330] T 640fdae70a5b460bb321a69b68b1aa35 P fe48f25970b145629c3e36b146488cb0: stopping tablet replica
I20260929 20:22:31.148970 19288 raft_consensus.cc:2229] T 640fdae70a5b460bb321a69b68b1aa35 P fe48f25970b145629c3e36b146488cb0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 20:22:31.149506 19288 raft_consensus.cc:2258] T 640fdae70a5b460bb321a69b68b1aa35 P fe48f25970b145629c3e36b146488cb0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:22:31.150357 19290 tablet_replica.cc:330] T 640fdae70a5b460bb321a69b68b1aa35 P 97175f85e6784f6bb1c4f433104a3802: stopping tablet replica
I20260929 20:22:31.150573 19289 tablet_replica.cc:330] T 640fdae70a5b460bb321a69b68b1aa35 P 744ee3020a694d7fad3c0ee1c7e50a26: stopping tablet replica
I20260929 20:22:31.150805 19290 raft_consensus.cc:2229] T 640fdae70a5b460bb321a69b68b1aa35 P 97175f85e6784f6bb1c4f433104a3802 [term 1 LEADER]: Raft consensus shutting down.
I20260929 20:22:31.151093 19289 raft_consensus.cc:2229] T 640fdae70a5b460bb321a69b68b1aa35 P 744ee3020a694d7fad3c0ee1c7e50a26 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 20:22:31.151607 19289 raft_consensus.cc:2258] T 640fdae70a5b460bb321a69b68b1aa35 P 744ee3020a694d7fad3c0ee1c7e50a26 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:22:31.151604 19290 raft_consensus.cc:2258] T 640fdae70a5b460bb321a69b68b1aa35 P 97175f85e6784f6bb1c4f433104a3802 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:22:31.153741 19288 ts_tablet_manager.cc:1905] T 640fdae70a5b460bb321a69b68b1aa35 P fe48f25970b145629c3e36b146488cb0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:22:31.155045 18893 tablet_service.cc:1467] Processing CreateTablet for tablet bd1284f0dede47d98c087b6e948637af (DEFAULT_TABLE table=test-table [id=fae0c4a2f57641c5bd185c7d460f5180]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:22:31.155174 19034 tablet_service.cc:1467] Processing CreateTablet for tablet bd1284f0dede47d98c087b6e948637af (DEFAULT_TABLE table=test-table [id=fae0c4a2f57641c5bd185c7d460f5180]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:22:31.155562 18893 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd1284f0dede47d98c087b6e948637af. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:22:31.155660 19034 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd1284f0dede47d98c087b6e948637af. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:22:31.155916 19174 tablet_service.cc:1467] Processing CreateTablet for tablet bd1284f0dede47d98c087b6e948637af (DEFAULT_TABLE table=test-table [id=fae0c4a2f57641c5bd185c7d460f5180]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:22:31.156111 19290 ts_tablet_manager.cc:1905] T 640fdae70a5b460bb321a69b68b1aa35 P 97175f85e6784f6bb1c4f433104a3802: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:22:31.158154 19289 ts_tablet_manager.cc:1905] T 640fdae70a5b460bb321a69b68b1aa35 P 744ee3020a694d7fad3c0ee1c7e50a26: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:22:31.159251 19174 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd1284f0dede47d98c087b6e948637af. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:22:31.163111 19290 ts_tablet_manager.cc:1918] T 640fdae70a5b460bb321a69b68b1aa35 P 97175f85e6784f6bb1c4f433104a3802: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 20:22:31.163286 19290 log.cc:1196] T 640fdae70a5b460bb321a69b68b1aa35 P 97175f85e6784f6bb1c4f433104a3802: Deleting WAL directory at /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790713339161492-17621-0/minicluster-data/ts-1/wal/wals/640fdae70a5b460bb321a69b68b1aa35
I20260929 20:22:31.163865 19290 ts_tablet_manager.cc:1939] T 640fdae70a5b460bb321a69b68b1aa35 P 97175f85e6784f6bb1c4f433104a3802: Deleting consensus metadata
I20260929 20:22:31.165848 19265 tablet_bootstrap.cc:492] T bd1284f0dede47d98c087b6e948637af P fe48f25970b145629c3e36b146488cb0: Bootstrap starting.
I20260929 20:22:31.166460 19264 tablet_bootstrap.cc:492] T bd1284f0dede47d98c087b6e948637af P 744ee3020a694d7fad3c0ee1c7e50a26: Bootstrap starting.
I20260929 20:22:31.167615 19265 tablet_bootstrap.cc:654] T bd1284f0dede47d98c087b6e948637af P fe48f25970b145629c3e36b146488cb0: Neither blocks nor log segments found. Creating new log.
I20260929 20:22:31.168206 19264 tablet_bootstrap.cc:654] T bd1284f0dede47d98c087b6e948637af P 744ee3020a694d7fad3c0ee1c7e50a26: Neither blocks nor log segments found. Creating new log.
I20260929 20:22:31.170019 19288 ts_tablet_manager.cc:1918] T 640fdae70a5b460bb321a69b68b1aa35 P fe48f25970b145629c3e36b146488cb0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 20:22:31.170303 19288 log.cc:1196] T 640fdae70a5b460bb321a69b68b1aa35 P fe48f25970b145629c3e36b146488cb0: Deleting WAL directory at /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790713339161492-17621-0/minicluster-data/ts-2/wal/wals/640fdae70a5b460bb321a69b68b1aa35
I20260929 20:22:31.171034 19288 ts_tablet_manager.cc:1939] T 640fdae70a5b460bb321a69b68b1aa35 P fe48f25970b145629c3e36b146488cb0: Deleting consensus metadata
I20260929 20:22:31.173314 18739 catalog_manager.cc:4841] TS fe48f25970b145629c3e36b146488cb0 (127.17.53.67:34979): tablet 640fdae70a5b460bb321a69b68b1aa35 (table test-table [id=ff77cc6bd5014ebd9b88fe6588450366]) successfully deleted
I20260929 20:22:31.175246 18740 catalog_manager.cc:4841] TS 97175f85e6784f6bb1c4f433104a3802 (127.17.53.66:46249): tablet 640fdae70a5b460bb321a69b68b1aa35 (table test-table [id=ff77cc6bd5014ebd9b88fe6588450366]) successfully deleted
I20260929 20:22:31.175750 19264 tablet_bootstrap.cc:492] T bd1284f0dede47d98c087b6e948637af P 744ee3020a694d7fad3c0ee1c7e50a26: No bootstrap required, opened a new log
I20260929 20:22:31.175887 19264 ts_tablet_manager.cc:1397] T bd1284f0dede47d98c087b6e948637af P 744ee3020a694d7fad3c0ee1c7e50a26: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.004s
I20260929 20:22:31.176672 19264 raft_consensus.cc:348] T bd1284f0dede47d98c087b6e948637af P 744ee3020a694d7fad3c0ee1c7e50a26 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "744ee3020a694d7fad3c0ee1c7e50a26" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 37593 } } peers { permanent_uuid: "97175f85e6784f6bb1c4f433104a3802" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 46249 } } peers { permanent_uuid: "fe48f25970b145629c3e36b146488cb0" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 34979 } }
I20260929 20:22:31.176837 19268 tablet_bootstrap.cc:492] T bd1284f0dede47d98c087b6e948637af P 97175f85e6784f6bb1c4f433104a3802: Bootstrap starting.
I20260929 20:22:31.176891 19264 raft_consensus.cc:374] T bd1284f0dede47d98c087b6e948637af P 744ee3020a694d7fad3c0ee1c7e50a26 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:22:31.176986 19264 raft_consensus.cc:729] T bd1284f0dede47d98c087b6e948637af P 744ee3020a694d7fad3c0ee1c7e50a26 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 744ee3020a694d7fad3c0ee1c7e50a26, State: Initialized, Role: FOLLOWER
I20260929 20:22:31.177742 19289 ts_tablet_manager.cc:1918] T 640fdae70a5b460bb321a69b68b1aa35 P 744ee3020a694d7fad3c0ee1c7e50a26: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 20:22:31.177968 19289 log.cc:1196] T 640fdae70a5b460bb321a69b68b1aa35 P 744ee3020a694d7fad3c0ee1c7e50a26: Deleting WAL directory at /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790713339161492-17621-0/minicluster-data/ts-0/wal/wals/640fdae70a5b460bb321a69b68b1aa35
I20260929 20:22:31.178570 19289 ts_tablet_manager.cc:1939] T 640fdae70a5b460bb321a69b68b1aa35 P 744ee3020a694d7fad3c0ee1c7e50a26: Deleting consensus metadata
I20260929 20:22:31.178617 19268 tablet_bootstrap.cc:654] T bd1284f0dede47d98c087b6e948637af P 97175f85e6784f6bb1c4f433104a3802: Neither blocks nor log segments found. Creating new log.
I20260929 20:22:31.177335 19264 consensus_queue.cc:260] T bd1284f0dede47d98c087b6e948637af P 744ee3020a694d7fad3c0ee1c7e50a26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "744ee3020a694d7fad3c0ee1c7e50a26" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 37593 } } peers { permanent_uuid: "97175f85e6784f6bb1c4f433104a3802" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 46249 } } peers { permanent_uuid: "fe48f25970b145629c3e36b146488cb0" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 34979 } }
I20260929 20:22:31.179104 19264 ts_tablet_manager.cc:1428] T bd1284f0dede47d98c087b6e948637af P 744ee3020a694d7fad3c0ee1c7e50a26: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260929 20:22:31.180312 19265 tablet_bootstrap.cc:492] T bd1284f0dede47d98c087b6e948637af P fe48f25970b145629c3e36b146488cb0: No bootstrap required, opened a new log
I20260929 20:22:31.180526 19265 ts_tablet_manager.cc:1397] T bd1284f0dede47d98c087b6e948637af P fe48f25970b145629c3e36b146488cb0: Time spent bootstrapping tablet: real 0.015s user 0.007s sys 0.003s
I20260929 20:22:31.181484 19265 raft_consensus.cc:348] T bd1284f0dede47d98c087b6e948637af P fe48f25970b145629c3e36b146488cb0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "744ee3020a694d7fad3c0ee1c7e50a26" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 37593 } } peers { permanent_uuid: "97175f85e6784f6bb1c4f433104a3802" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 46249 } } peers { permanent_uuid: "fe48f25970b145629c3e36b146488cb0" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 34979 } }
I20260929 20:22:31.181725 19265 raft_consensus.cc:374] T bd1284f0dede47d98c087b6e948637af P fe48f25970b145629c3e36b146488cb0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:22:31.181860 19265 raft_consensus.cc:729] T bd1284f0dede47d98c087b6e948637af P fe48f25970b145629c3e36b146488cb0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe48f25970b145629c3e36b146488cb0, State: Initialized, Role: FOLLOWER
I20260929 20:22:31.182195 19265 consensus_queue.cc:260] T bd1284f0dede47d98c087b6e948637af P fe48f25970b145629c3e36b146488cb0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "744ee3020a694d7fad3c0ee1c7e50a26" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 37593 } } peers { permanent_uuid: "97175f85e6784f6bb1c4f433104a3802" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 46249 } } peers { permanent_uuid: "fe48f25970b145629c3e36b146488cb0" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 34979 } }
I20260929 20:22:31.182926 19265 ts_tablet_manager.cc:1428] T bd1284f0dede47d98c087b6e948637af P fe48f25970b145629c3e36b146488cb0: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260929 20:22:31.183895 18737 catalog_manager.cc:4841] TS 744ee3020a694d7fad3c0ee1c7e50a26 (127.17.53.65:37593): tablet 640fdae70a5b460bb321a69b68b1aa35 (table test-table [id=ff77cc6bd5014ebd9b88fe6588450366]) successfully deleted
I20260929 20:22:31.186741 19268 tablet_bootstrap.cc:492] T bd1284f0dede47d98c087b6e948637af P 97175f85e6784f6bb1c4f433104a3802: No bootstrap required, opened a new log
I20260929 20:22:31.186949 19268 ts_tablet_manager.cc:1397] T bd1284f0dede47d98c087b6e948637af P 97175f85e6784f6bb1c4f433104a3802: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.003s
I20260929 20:22:31.187630 19268 raft_consensus.cc:348] T bd1284f0dede47d98c087b6e948637af P 97175f85e6784f6bb1c4f433104a3802 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "744ee3020a694d7fad3c0ee1c7e50a26" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 37593 } } peers { permanent_uuid: "97175f85e6784f6bb1c4f433104a3802" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 46249 } } peers { permanent_uuid: "fe48f25970b145629c3e36b146488cb0" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 34979 } }
I20260929 20:22:31.187810 19268 raft_consensus.cc:374] T bd1284f0dede47d98c087b6e948637af P 97175f85e6784f6bb1c4f433104a3802 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:22:31.187942 19268 raft_consensus.cc:729] T bd1284f0dede47d98c087b6e948637af P 97175f85e6784f6bb1c4f433104a3802 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97175f85e6784f6bb1c4f433104a3802, State: Initialized, Role: FOLLOWER
I20260929 20:22:31.188207 19268 consensus_queue.cc:260] T bd1284f0dede47d98c087b6e948637af P 97175f85e6784f6bb1c4f433104a3802 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "744ee3020a694d7fad3c0ee1c7e50a26" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 37593 } } peers { permanent_uuid: "97175f85e6784f6bb1c4f433104a3802" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 46249 } } peers { permanent_uuid: "fe48f25970b145629c3e36b146488cb0" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 34979 } }
I20260929 20:22:31.189389 19268 ts_tablet_manager.cc:1428] T bd1284f0dede47d98c087b6e948637af P 97175f85e6784f6bb1c4f433104a3802: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260929 20:22:31.189508 19272 raft_consensus.cc:482] T bd1284f0dede47d98c087b6e948637af P 97175f85e6784f6bb1c4f433104a3802 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 20:22:31.189657 19272 raft_consensus.cc:504] T bd1284f0dede47d98c087b6e948637af P 97175f85e6784f6bb1c4f433104a3802 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "744ee3020a694d7fad3c0ee1c7e50a26" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 37593 } } peers { permanent_uuid: "97175f85e6784f6bb1c4f433104a3802" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 46249 } } peers { permanent_uuid: "fe48f25970b145629c3e36b146488cb0" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 34979 } }
I20260929 20:22:31.190310 19272 leader_election.cc:288] T bd1284f0dede47d98c087b6e948637af P 97175f85e6784f6bb1c4f433104a3802 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 744ee3020a694d7fad3c0ee1c7e50a26 (127.17.53.65:37593), fe48f25970b145629c3e36b146488cb0 (127.17.53.67:34979)
I20260929 20:22:31.190711 19194 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd1284f0dede47d98c087b6e948637af" candidate_uuid: "97175f85e6784f6bb1c4f433104a3802" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe48f25970b145629c3e36b146488cb0" is_pre_election: true
I20260929 20:22:31.190721 18913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd1284f0dede47d98c087b6e948637af" candidate_uuid: "97175f85e6784f6bb1c4f433104a3802" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "744ee3020a694d7fad3c0ee1c7e50a26" is_pre_election: true
I20260929 20:22:31.190955 19194 raft_consensus.cc:2454] T bd1284f0dede47d98c087b6e948637af P fe48f25970b145629c3e36b146488cb0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97175f85e6784f6bb1c4f433104a3802 in term 0.
I20260929 20:22:31.191004 18913 raft_consensus.cc:2454] T bd1284f0dede47d98c087b6e948637af P 744ee3020a694d7fad3c0ee1c7e50a26 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97175f85e6784f6bb1c4f433104a3802 in term 0.
I20260929 20:22:31.191423 18986 leader_election.cc:302] T bd1284f0dede47d98c087b6e948637af P 97175f85e6784f6bb1c4f433104a3802 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 744ee3020a694d7fad3c0ee1c7e50a26, 97175f85e6784f6bb1c4f433104a3802, fe48f25970b145629c3e36b146488cb0; no voters:
I20260929 20:22:31.191736 19272 raft_consensus.cc:2781] T bd1284f0dede47d98c087b6e948637af P 97175f85e6784f6bb1c4f433104a3802 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 20:22:31.191869 19272 raft_consensus.cc:482] T bd1284f0dede47d98c087b6e948637af P 97175f85e6784f6bb1c4f433104a3802 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 20:22:31.191970 19272 raft_consensus.cc:3037] T bd1284f0dede47d98c087b6e948637af P 97175f85e6784f6bb1c4f433104a3802 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:22:31.195034 19272 raft_consensus.cc:504] T bd1284f0dede47d98c087b6e948637af P 97175f85e6784f6bb1c4f433104a3802 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "744ee3020a694d7fad3c0ee1c7e50a26" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 37593 } } peers { permanent_uuid: "97175f85e6784f6bb1c4f433104a3802" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 46249 } } peers { permanent_uuid: "fe48f25970b145629c3e36b146488cb0" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 34979 } }
I20260929 20:22:31.195704 19272 leader_election.cc:288] T bd1284f0dede47d98c087b6e948637af P 97175f85e6784f6bb1c4f433104a3802 [CANDIDATE]: Term 1 election: Requested vote from peers 744ee3020a694d7fad3c0ee1c7e50a26 (127.17.53.65:37593), fe48f25970b145629c3e36b146488cb0 (127.17.53.67:34979)
I20260929 20:22:31.196157 18913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd1284f0dede47d98c087b6e948637af" candidate_uuid: "97175f85e6784f6bb1c4f433104a3802" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "744ee3020a694d7fad3c0ee1c7e50a26"
I20260929 20:22:31.196372 18913 raft_consensus.cc:3037] T bd1284f0dede47d98c087b6e948637af P 744ee3020a694d7fad3c0ee1c7e50a26 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:22:31.198577 18913 raft_consensus.cc:2454] T bd1284f0dede47d98c087b6e948637af P 744ee3020a694d7fad3c0ee1c7e50a26 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97175f85e6784f6bb1c4f433104a3802 in term 1.
I20260929 20:22:31.199148 18984 leader_election.cc:302] T bd1284f0dede47d98c087b6e948637af P 97175f85e6784f6bb1c4f433104a3802 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 744ee3020a694d7fad3c0ee1c7e50a26, 97175f85e6784f6bb1c4f433104a3802; no voters:
I20260929 20:22:31.199344 19194 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd1284f0dede47d98c087b6e948637af" candidate_uuid: "97175f85e6784f6bb1c4f433104a3802" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe48f25970b145629c3e36b146488cb0"
I20260929 20:22:31.199487 19272 raft_consensus.cc:2781] T bd1284f0dede47d98c087b6e948637af P 97175f85e6784f6bb1c4f433104a3802 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:22:31.199558 19194 raft_consensus.cc:3037] T bd1284f0dede47d98c087b6e948637af P fe48f25970b145629c3e36b146488cb0 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:22:31.199635 19272 raft_consensus.cc:686] T bd1284f0dede47d98c087b6e948637af P 97175f85e6784f6bb1c4f433104a3802 [term 1 LEADER]: Becoming Leader. State: Replica: 97175f85e6784f6bb1c4f433104a3802, State: Running, Role: LEADER
I20260929 20:22:31.199901 19272 consensus_queue.cc:237] T bd1284f0dede47d98c087b6e948637af P 97175f85e6784f6bb1c4f433104a3802 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "744ee3020a694d7fad3c0ee1c7e50a26" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 37593 } } peers { permanent_uuid: "97175f85e6784f6bb1c4f433104a3802" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 46249 } } peers { permanent_uuid: "fe48f25970b145629c3e36b146488cb0" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 34979 } }
I20260929 20:22:31.202270 19194 raft_consensus.cc:2454] T bd1284f0dede47d98c087b6e948637af P fe48f25970b145629c3e36b146488cb0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97175f85e6784f6bb1c4f433104a3802 in term 1.
I20260929 20:22:31.202519 18753 catalog_manager.cc:5495] T bd1284f0dede47d98c087b6e948637af P 97175f85e6784f6bb1c4f433104a3802 reported cstate change: term changed from 0 to 1, leader changed from <none> to 97175f85e6784f6bb1c4f433104a3802 (127.17.53.66). New cstate: current_term: 1 leader_uuid: "97175f85e6784f6bb1c4f433104a3802" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "744ee3020a694d7fad3c0ee1c7e50a26" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 37593 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "97175f85e6784f6bb1c4f433104a3802" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 46249 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fe48f25970b145629c3e36b146488cb0" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 34979 } health_report { overall_health: UNKNOWN } } }
Sep 29 20:22:31 dist-test-slave-zbnc krb5kdc[18708](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713349, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.53.126@KRBTEST.COM for kudu/127.17.53.126@KRBTEST.COM
I20260929 20:22:31.260450 18850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:31.245584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49108 (local address 127.17.53.126:43983)
0929 20:22:31.245789 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0929 20:22:31.245793 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:22:31.246469 (+ 676us) server_negotiation.cc:407] Connection header received
0929 20:22:31.246613 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:22:31.246618 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:22:31.246713 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:22:31.246829 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:22:31.248020 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:31.249147 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:31.250577 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:31.251310 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:31.251438 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:22:31.254959 (+ 3521us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:22:31.254964 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:22:31.254967 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:22:31.254994 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:22:31.257644 (+ 2650us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:22:31.258488 (+ 844us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:22:31.258492 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:22:31.258494 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:22:31.258613 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:22:31.259090 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:22:31.259097 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:22:31.259103 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:22:31.259416 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:22:31.259525 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0929 20:22:31.260026 (+ 501us) server_negotiation.cc:299] Negotiation successful
0929 20:22:31.260187 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260929 20:22:31.260850 19300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:31.245717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:43983 (local address 127.0.0.1:49108)
0929 20:22:31.246355 (+ 638us) negotiation.cc:107] Waiting for socket to connect
0929 20:22:31.246371 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 20:22:31.246495 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:22:31.246913 (+ 418us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:22:31.246924 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:22:31.247605 (+ 681us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:22:31.247867 (+ 262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:22:31.247876 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:22:31.249268 (+ 1392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:22:31.249271 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:22:31.250431 (+ 1160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:22:31.250443 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:22:31.251478 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:22:31.251482 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:22:31.251593 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:22:31.252279 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:22:31.252295 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:22:31.254725 (+ 2430us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:22:31.257774 (+ 3049us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:22:31.257790 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:22:31.257799 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:22:31.258383 (+ 584us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:22:31.258718 (+ 335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:22:31.258723 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:22:31.258728 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:22:31.258935 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:22:31.259560 (+ 625us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:22:31.259570 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:22:31.259761 (+ 191us) client_negotiation.cc:769] Sending connection context
0929 20:22:31.259952 (+ 191us) client_negotiation.cc:240] Negotiation successful
0929 20:22:31.260371 (+ 419us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":519,"mutex_wait_us":117,"thread_start_us":358,"threads_started":1}
I20260929 20:22:31.265786 18753 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.17.53.126@KRBTEST.COM'} at 127.0.0.1:49108:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260929 20:22:31.267186 18753 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 20:22:31.273813 18893 tablet_service.cc:1467] Processing CreateTablet for tablet f3191d83a7e344bd8487b0e21a516e9f (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=d7736e4461b0414b87f5dbf693934af7]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 20:22:31.274351 18893 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f3191d83a7e344bd8487b0e21a516e9f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:22:31.274158 19174 tablet_service.cc:1467] Processing CreateTablet for tablet f3191d83a7e344bd8487b0e21a516e9f (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=d7736e4461b0414b87f5dbf693934af7]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 20:22:31.274693 19174 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f3191d83a7e344bd8487b0e21a516e9f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:22:31.279651 19264 tablet_bootstrap.cc:492] T f3191d83a7e344bd8487b0e21a516e9f P 744ee3020a694d7fad3c0ee1c7e50a26: Bootstrap starting.
I20260929 20:22:31.280206 19034 tablet_service.cc:1467] Processing CreateTablet for tablet f3191d83a7e344bd8487b0e21a516e9f (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=d7736e4461b0414b87f5dbf693934af7]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 20:22:31.280788 19034 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f3191d83a7e344bd8487b0e21a516e9f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:22:31.281355 19264 tablet_bootstrap.cc:654] T f3191d83a7e344bd8487b0e21a516e9f P 744ee3020a694d7fad3c0ee1c7e50a26: Neither blocks nor log segments found. Creating new log.
I20260929 20:22:31.282701 19265 tablet_bootstrap.cc:492] T f3191d83a7e344bd8487b0e21a516e9f P fe48f25970b145629c3e36b146488cb0: Bootstrap starting.
I20260929 20:22:31.284967 19265 tablet_bootstrap.cc:654] T f3191d83a7e344bd8487b0e21a516e9f P fe48f25970b145629c3e36b146488cb0: Neither blocks nor log segments found. Creating new log.
I20260929 20:22:31.291563 19264 tablet_bootstrap.cc:492] T f3191d83a7e344bd8487b0e21a516e9f P 744ee3020a694d7fad3c0ee1c7e50a26: No bootstrap required, opened a new log
I20260929 20:22:31.291791 19264 ts_tablet_manager.cc:1397] T f3191d83a7e344bd8487b0e21a516e9f P 744ee3020a694d7fad3c0ee1c7e50a26: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.008s
I20260929 20:22:31.291805 19265 tablet_bootstrap.cc:492] T f3191d83a7e344bd8487b0e21a516e9f P fe48f25970b145629c3e36b146488cb0: No bootstrap required, opened a new log
I20260929 20:22:31.291960 19265 ts_tablet_manager.cc:1397] T f3191d83a7e344bd8487b0e21a516e9f P fe48f25970b145629c3e36b146488cb0: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.000s
I20260929 20:22:31.292701 19265 raft_consensus.cc:348] T f3191d83a7e344bd8487b0e21a516e9f P fe48f25970b145629c3e36b146488cb0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "744ee3020a694d7fad3c0ee1c7e50a26" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 37593 } } peers { permanent_uuid: "97175f85e6784f6bb1c4f433104a3802" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 46249 } } peers { permanent_uuid: "fe48f25970b145629c3e36b146488cb0" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 34979 } }
I20260929 20:22:31.292886 19268 tablet_bootstrap.cc:492] T f3191d83a7e344bd8487b0e21a516e9f P 97175f85e6784f6bb1c4f433104a3802: Bootstrap starting.
I20260929 20:22:31.292959 19265 raft_consensus.cc:374] T f3191d83a7e344bd8487b0e21a516e9f P fe48f25970b145629c3e36b146488cb0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:22:31.292735 19264 raft_consensus.cc:348] T f3191d83a7e344bd8487b0e21a516e9f P 744ee3020a694d7fad3c0ee1c7e50a26 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "744ee3020a694d7fad3c0ee1c7e50a26" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 37593 } } peers { permanent_uuid: "97175f85e6784f6bb1c4f433104a3802" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 46249 } } peers { permanent_uuid: "fe48f25970b145629c3e36b146488cb0" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 34979 } }
I20260929 20:22:31.293058 19265 raft_consensus.cc:729] T f3191d83a7e344bd8487b0e21a516e9f P fe48f25970b145629c3e36b146488cb0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe48f25970b145629c3e36b146488cb0, State: Initialized, Role: FOLLOWER
I20260929 20:22:31.293357 19265 consensus_queue.cc:260] T f3191d83a7e344bd8487b0e21a516e9f P fe48f25970b145629c3e36b146488cb0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "744ee3020a694d7fad3c0ee1c7e50a26" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 37593 } } peers { permanent_uuid: "97175f85e6784f6bb1c4f433104a3802" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 46249 } } peers { permanent_uuid: "fe48f25970b145629c3e36b146488cb0" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 34979 } }
I20260929 20:22:31.293916 19265 ts_tablet_manager.cc:1428] T f3191d83a7e344bd8487b0e21a516e9f P fe48f25970b145629c3e36b146488cb0: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260929 20:22:31.293855 19270 tablet_replica.cc:439] T f3191d83a7e344bd8487b0e21a516e9f P fe48f25970b145629c3e36b146488cb0: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "744ee3020a694d7fad3c0ee1c7e50a26" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 37593 } } peers { permanent_uuid: "97175f85e6784f6bb1c4f433104a3802" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 46249 } } peers { permanent_uuid: "fe48f25970b145629c3e36b146488cb0" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 34979 } } }
I20260929 20:22:31.294214 19270 tablet_replica.cc:442] T f3191d83a7e344bd8487b0e21a516e9f P fe48f25970b145629c3e36b146488cb0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 20:22:31.294592 19264 raft_consensus.cc:374] T f3191d83a7e344bd8487b0e21a516e9f P 744ee3020a694d7fad3c0ee1c7e50a26 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:22:31.294677 19268 tablet_bootstrap.cc:654] T f3191d83a7e344bd8487b0e21a516e9f P 97175f85e6784f6bb1c4f433104a3802: Neither blocks nor log segments found. Creating new log.
I20260929 20:22:31.294701 19264 raft_consensus.cc:729] T f3191d83a7e344bd8487b0e21a516e9f P 744ee3020a694d7fad3c0ee1c7e50a26 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 744ee3020a694d7fad3c0ee1c7e50a26, State: Initialized, Role: FOLLOWER
I20260929 20:22:31.294963 19264 consensus_queue.cc:260] T f3191d83a7e344bd8487b0e21a516e9f P 744ee3020a694d7fad3c0ee1c7e50a26 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "744ee3020a694d7fad3c0ee1c7e50a26" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 37593 } } peers { permanent_uuid: "97175f85e6784f6bb1c4f433104a3802" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 46249 } } peers { permanent_uuid: "fe48f25970b145629c3e36b146488cb0" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 34979 } }
I20260929 20:22:31.295590 19264 ts_tablet_manager.cc:1428] T f3191d83a7e344bd8487b0e21a516e9f P 744ee3020a694d7fad3c0ee1c7e50a26: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20260929 20:22:31.295472 19271 tablet_replica.cc:439] T f3191d83a7e344bd8487b0e21a516e9f P 744ee3020a694d7fad3c0ee1c7e50a26: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "744ee3020a694d7fad3c0ee1c7e50a26" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 37593 } } peers { permanent_uuid: "97175f85e6784f6bb1c4f433104a3802" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 46249 } } peers { permanent_uuid: "fe48f25970b145629c3e36b146488cb0" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 34979 } } }
I20260929 20:22:31.295877 19271 tablet_replica.cc:442] T f3191d83a7e344bd8487b0e21a516e9f P 744ee3020a694d7fad3c0ee1c7e50a26: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 20:22:31.298310 19268 tablet_bootstrap.cc:492] T f3191d83a7e344bd8487b0e21a516e9f P 97175f85e6784f6bb1c4f433104a3802: No bootstrap required, opened a new log
I20260929 20:22:31.298535 19268 ts_tablet_manager.cc:1397] T f3191d83a7e344bd8487b0e21a516e9f P 97175f85e6784f6bb1c4f433104a3802: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260929 20:22:31.299429 19268 raft_consensus.cc:348] T f3191d83a7e344bd8487b0e21a516e9f P 97175f85e6784f6bb1c4f433104a3802 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "744ee3020a694d7fad3c0ee1c7e50a26" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 37593 } } peers { permanent_uuid: "97175f85e6784f6bb1c4f433104a3802" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 46249 } } peers { permanent_uuid: "fe48f25970b145629c3e36b146488cb0" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 34979 } }
I20260929 20:22:31.299695 19268 raft_consensus.cc:374] T f3191d83a7e344bd8487b0e21a516e9f P 97175f85e6784f6bb1c4f433104a3802 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:22:31.299817 19268 raft_consensus.cc:729] T f3191d83a7e344bd8487b0e21a516e9f P 97175f85e6784f6bb1c4f433104a3802 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97175f85e6784f6bb1c4f433104a3802, State: Initialized, Role: FOLLOWER
I20260929 20:22:31.300096 19268 consensus_queue.cc:260] T f3191d83a7e344bd8487b0e21a516e9f P 97175f85e6784f6bb1c4f433104a3802 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "744ee3020a694d7fad3c0ee1c7e50a26" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 37593 } } peers { permanent_uuid: "97175f85e6784f6bb1c4f433104a3802" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 46249 } } peers { permanent_uuid: "fe48f25970b145629c3e36b146488cb0" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 34979 } }
I20260929 20:22:31.300690 19268 ts_tablet_manager.cc:1428] T f3191d83a7e344bd8487b0e21a516e9f P 97175f85e6784f6bb1c4f433104a3802: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260929 20:22:31.300510 19272 tablet_replica.cc:439] T f3191d83a7e344bd8487b0e21a516e9f P 97175f85e6784f6bb1c4f433104a3802: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "744ee3020a694d7fad3c0ee1c7e50a26" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 37593 } } peers { permanent_uuid: "97175f85e6784f6bb1c4f433104a3802" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 46249 } } peers { permanent_uuid: "fe48f25970b145629c3e36b146488cb0" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 34979 } } }
I20260929 20:22:31.300971 19272 tablet_replica.cc:442] T f3191d83a7e344bd8487b0e21a516e9f P 97175f85e6784f6bb1c4f433104a3802: This TxnStatusTablet replica's current role is: FOLLOWER
W20260929 20:22:31.366459 19246 tablet.cc:2369] T bd1284f0dede47d98c087b6e948637af P fe48f25970b145629c3e36b146488cb0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 20:22:31.600076 19272 raft_consensus.cc:482] T f3191d83a7e344bd8487b0e21a516e9f P 97175f85e6784f6bb1c4f433104a3802 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 20:22:31.600283 19272 raft_consensus.cc:504] T f3191d83a7e344bd8487b0e21a516e9f P 97175f85e6784f6bb1c4f433104a3802 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "744ee3020a694d7fad3c0ee1c7e50a26" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 37593 } } peers { permanent_uuid: "97175f85e6784f6bb1c4f433104a3802" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 46249 } } peers { permanent_uuid: "fe48f25970b145629c3e36b146488cb0" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 34979 } }
I20260929 20:22:31.600900 19272 leader_election.cc:288] T f3191d83a7e344bd8487b0e21a516e9f P 97175f85e6784f6bb1c4f433104a3802 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 744ee3020a694d7fad3c0ee1c7e50a26 (127.17.53.65:37593), fe48f25970b145629c3e36b146488cb0 (127.17.53.67:34979)
I20260929 20:22:31.601534 19194 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3191d83a7e344bd8487b0e21a516e9f" candidate_uuid: "97175f85e6784f6bb1c4f433104a3802" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe48f25970b145629c3e36b146488cb0" is_pre_election: true
I20260929 20:22:31.601533 18913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3191d83a7e344bd8487b0e21a516e9f" candidate_uuid: "97175f85e6784f6bb1c4f433104a3802" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "744ee3020a694d7fad3c0ee1c7e50a26" is_pre_election: true
I20260929 20:22:31.601795 19194 raft_consensus.cc:2454] T f3191d83a7e344bd8487b0e21a516e9f P fe48f25970b145629c3e36b146488cb0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97175f85e6784f6bb1c4f433104a3802 in term 0.
I20260929 20:22:31.601828 18913 raft_consensus.cc:2454] T f3191d83a7e344bd8487b0e21a516e9f P 744ee3020a694d7fad3c0ee1c7e50a26 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97175f85e6784f6bb1c4f433104a3802 in term 0.
I20260929 20:22:31.602234 18984 leader_election.cc:302] T f3191d83a7e344bd8487b0e21a516e9f P 97175f85e6784f6bb1c4f433104a3802 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 744ee3020a694d7fad3c0ee1c7e50a26, 97175f85e6784f6bb1c4f433104a3802; no voters:
I20260929 20:22:31.602532 19272 raft_consensus.cc:2781] T f3191d83a7e344bd8487b0e21a516e9f P 97175f85e6784f6bb1c4f433104a3802 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 20:22:31.602625 19272 raft_consensus.cc:482] T f3191d83a7e344bd8487b0e21a516e9f P 97175f85e6784f6bb1c4f433104a3802 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 20:22:31.602756 19272 raft_consensus.cc:3037] T f3191d83a7e344bd8487b0e21a516e9f P 97175f85e6784f6bb1c4f433104a3802 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:22:31.604799 19272 raft_consensus.cc:504] T f3191d83a7e344bd8487b0e21a516e9f P 97175f85e6784f6bb1c4f433104a3802 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "744ee3020a694d7fad3c0ee1c7e50a26" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 37593 } } peers { permanent_uuid: "97175f85e6784f6bb1c4f433104a3802" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 46249 } } peers { permanent_uuid: "fe48f25970b145629c3e36b146488cb0" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 34979 } }
I20260929 20:22:31.605415 19272 leader_election.cc:288] T f3191d83a7e344bd8487b0e21a516e9f P 97175f85e6784f6bb1c4f433104a3802 [CANDIDATE]: Term 1 election: Requested vote from peers 744ee3020a694d7fad3c0ee1c7e50a26 (127.17.53.65:37593), fe48f25970b145629c3e36b146488cb0 (127.17.53.67:34979)
I20260929 20:22:31.605897 18913 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3191d83a7e344bd8487b0e21a516e9f" candidate_uuid: "97175f85e6784f6bb1c4f433104a3802" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "744ee3020a694d7fad3c0ee1c7e50a26"
I20260929 20:22:31.605897 19194 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f3191d83a7e344bd8487b0e21a516e9f" candidate_uuid: "97175f85e6784f6bb1c4f433104a3802" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe48f25970b145629c3e36b146488cb0"
I20260929 20:22:31.606110 18913 raft_consensus.cc:3037] T f3191d83a7e344bd8487b0e21a516e9f P 744ee3020a694d7fad3c0ee1c7e50a26 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:22:31.606154 19194 raft_consensus.cc:3037] T f3191d83a7e344bd8487b0e21a516e9f P fe48f25970b145629c3e36b146488cb0 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:22:31.608124 19272 consensus_queue.cc:1035] T bd1284f0dede47d98c087b6e948637af P 97175f85e6784f6bb1c4f433104a3802 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe48f25970b145629c3e36b146488cb0" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 34979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:22:31.608771 18913 raft_consensus.cc:2454] T f3191d83a7e344bd8487b0e21a516e9f P 744ee3020a694d7fad3c0ee1c7e50a26 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97175f85e6784f6bb1c4f433104a3802 in term 1.
I20260929 20:22:31.608829 19194 raft_consensus.cc:2454] T f3191d83a7e344bd8487b0e21a516e9f P fe48f25970b145629c3e36b146488cb0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97175f85e6784f6bb1c4f433104a3802 in term 1.
I20260929 20:22:31.609205 18984 leader_election.cc:302] T f3191d83a7e344bd8487b0e21a516e9f P 97175f85e6784f6bb1c4f433104a3802 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 744ee3020a694d7fad3c0ee1c7e50a26, 97175f85e6784f6bb1c4f433104a3802; no voters:
I20260929 20:22:31.609594 19272 raft_consensus.cc:2781] T f3191d83a7e344bd8487b0e21a516e9f P 97175f85e6784f6bb1c4f433104a3802 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:22:31.609752 19272 raft_consensus.cc:686] T f3191d83a7e344bd8487b0e21a516e9f P 97175f85e6784f6bb1c4f433104a3802 [term 1 LEADER]: Becoming Leader. State: Replica: 97175f85e6784f6bb1c4f433104a3802, State: Running, Role: LEADER
I20260929 20:22:31.610019 19272 consensus_queue.cc:237] T f3191d83a7e344bd8487b0e21a516e9f P 97175f85e6784f6bb1c4f433104a3802 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "744ee3020a694d7fad3c0ee1c7e50a26" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 37593 } } peers { permanent_uuid: "97175f85e6784f6bb1c4f433104a3802" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 46249 } } peers { permanent_uuid: "fe48f25970b145629c3e36b146488cb0" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 34979 } }
I20260929 20:22:31.611515 19275 tablet_replica.cc:439] T f3191d83a7e344bd8487b0e21a516e9f P 97175f85e6784f6bb1c4f433104a3802: TxnStatusTablet state changed. Reason: New leader 97175f85e6784f6bb1c4f433104a3802. Latest consensus state: current_term: 1 leader_uuid: "97175f85e6784f6bb1c4f433104a3802" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "744ee3020a694d7fad3c0ee1c7e50a26" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 37593 } } peers { permanent_uuid: "97175f85e6784f6bb1c4f433104a3802" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 46249 } } peers { permanent_uuid: "fe48f25970b145629c3e36b146488cb0" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 34979 } } }
I20260929 20:22:31.611833 19275 tablet_replica.cc:442] T f3191d83a7e344bd8487b0e21a516e9f P 97175f85e6784f6bb1c4f433104a3802: This TxnStatusTablet replica's current role is: LEADER
I20260929 20:22:31.612808 19306 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260929 20:22:31.613072 19306 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260929 20:22:31.613544 18754 catalog_manager.cc:5495] T f3191d83a7e344bd8487b0e21a516e9f P 97175f85e6784f6bb1c4f433104a3802 reported cstate change: term changed from 0 to 1, leader changed from <none> to 97175f85e6784f6bb1c4f433104a3802 (127.17.53.66). New cstate: current_term: 1 leader_uuid: "97175f85e6784f6bb1c4f433104a3802" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "744ee3020a694d7fad3c0ee1c7e50a26" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 37593 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "97175f85e6784f6bb1c4f433104a3802" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 46249 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fe48f25970b145629c3e36b146488cb0" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 34979 } health_report { overall_health: UNKNOWN } } }
I20260929 20:22:31.615949 19275 consensus_queue.cc:1035] T bd1284f0dede47d98c087b6e948637af P 97175f85e6784f6bb1c4f433104a3802 [LEADER]: Connected to new peer: Peer: permanent_uuid: "744ee3020a694d7fad3c0ee1c7e50a26" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 37593 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:22:31.671340 19310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:31.664158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48824 (local address 127.17.53.66:46249)
0929 20:22:31.665901 (+ 1743us) server_negotiation.cc:206] Beginning negotiation
0929 20:22:31.665905 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:22:31.665921 (+ 16us) server_negotiation.cc:407] Connection header received
0929 20:22:31.665976 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:22:31.665979 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:22:31.666041 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:22:31.666121 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 20:22:31.666728 (+ 607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:31.667551 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:31.669470 (+ 1919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:31.669976 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:31.670069 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:22:31.670612 (+ 543us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 20:22:31.670735 (+ 123us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 20:22:31.670833 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0929 20:22:31.671018 (+ 185us) server_negotiation.cc:299] Negotiation successful
0929 20:22:31.671076 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1634,"thread_start_us":1485,"threads_started":1}
I20260929 20:22:31.671482 19300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:31.664078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.66:46249 (local address 127.0.0.1:48824)
0929 20:22:31.664278 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0929 20:22:31.664306 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0929 20:22:31.664415 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:22:31.666204 (+ 1789us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:22:31.666209 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:22:31.666223 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 20:22:31.666469 (+ 246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:22:31.666481 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:22:31.667679 (+ 1198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:22:31.667686 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:22:31.669323 (+ 1637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:22:31.669341 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:22:31.670138 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:22:31.670144 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:22:31.670366 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:22:31.670446 (+ 80us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 20:22:31.670875 (+ 429us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 20:22:31.670896 (+ 21us) client_negotiation.cc:769] Sending connection context
0929 20:22:31.670989 (+ 93us) client_negotiation.cc:240] Negotiation successful
0929 20:22:31.671152 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20260929 20:22:31.675269 18913 raft_consensus.cc:1261] T f3191d83a7e344bd8487b0e21a516e9f P 744ee3020a694d7fad3c0ee1c7e50a26 [term 1 FOLLOWER]: Refusing update from remote peer 97175f85e6784f6bb1c4f433104a3802: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:22:31.675251 19193 raft_consensus.cc:1261] T f3191d83a7e344bd8487b0e21a516e9f P fe48f25970b145629c3e36b146488cb0 [term 1 FOLLOWER]: Refusing update from remote peer 97175f85e6784f6bb1c4f433104a3802: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:22:31.676203 19283 consensus_queue.cc:1035] T f3191d83a7e344bd8487b0e21a516e9f P 97175f85e6784f6bb1c4f433104a3802 [LEADER]: Connected to new peer: Peer: permanent_uuid: "744ee3020a694d7fad3c0ee1c7e50a26" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 37593 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:22:31.676604 19275 consensus_queue.cc:1035] T f3191d83a7e344bd8487b0e21a516e9f P 97175f85e6784f6bb1c4f433104a3802 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe48f25970b145629c3e36b146488cb0" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 34979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:22:31.678659 19271 tablet_replica.cc:439] T f3191d83a7e344bd8487b0e21a516e9f P 744ee3020a694d7fad3c0ee1c7e50a26: TxnStatusTablet state changed. Reason: New leader 97175f85e6784f6bb1c4f433104a3802. Latest consensus state: current_term: 1 leader_uuid: "97175f85e6784f6bb1c4f433104a3802" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "744ee3020a694d7fad3c0ee1c7e50a26" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 37593 } } peers { permanent_uuid: "97175f85e6784f6bb1c4f433104a3802" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 46249 } } peers { permanent_uuid: "fe48f25970b145629c3e36b146488cb0" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 34979 } } }
I20260929 20:22:31.678921 19271 tablet_replica.cc:442] T f3191d83a7e344bd8487b0e21a516e9f P 744ee3020a694d7fad3c0ee1c7e50a26: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 20:22:31.680335 19278 mvcc.cc:205] Tried to move back new op lower bound from 7334761888455421952 to 7334761888197017600. Current Snapshot: MvccSnapshot[applied={T|T < 7334761888455421952}]
I20260929 20:22:31.681562 19279 mvcc.cc:205] Tried to move back new op lower bound from 7334761888455421952 to 7334761888197017600. Current Snapshot: MvccSnapshot[applied={T|T < 7334761888455421952}]
I20260929 20:22:31.685055 19271 tablet_replica.cc:439] T f3191d83a7e344bd8487b0e21a516e9f P 744ee3020a694d7fad3c0ee1c7e50a26: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "97175f85e6784f6bb1c4f433104a3802" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "744ee3020a694d7fad3c0ee1c7e50a26" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 37593 } } peers { permanent_uuid: "97175f85e6784f6bb1c4f433104a3802" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 46249 } } peers { permanent_uuid: "fe48f25970b145629c3e36b146488cb0" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 34979 } } }
I20260929 20:22:31.685320 19271 tablet_replica.cc:442] T f3191d83a7e344bd8487b0e21a516e9f P 744ee3020a694d7fad3c0ee1c7e50a26: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 20:22:31.689280 19275 tablet_replica.cc:439] T f3191d83a7e344bd8487b0e21a516e9f P 97175f85e6784f6bb1c4f433104a3802: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "97175f85e6784f6bb1c4f433104a3802" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "744ee3020a694d7fad3c0ee1c7e50a26" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 37593 } } peers { permanent_uuid: "97175f85e6784f6bb1c4f433104a3802" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 46249 } } peers { permanent_uuid: "fe48f25970b145629c3e36b146488cb0" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 34979 } } }
I20260929 20:22:31.689555 19275 tablet_replica.cc:442] T f3191d83a7e344bd8487b0e21a516e9f P 97175f85e6784f6bb1c4f433104a3802: This TxnStatusTablet replica's current role is: LEADER
I20260929 20:22:31.692303 19270 tablet_replica.cc:439] T f3191d83a7e344bd8487b0e21a516e9f P fe48f25970b145629c3e36b146488cb0: TxnStatusTablet state changed. Reason: New leader 97175f85e6784f6bb1c4f433104a3802. Latest consensus state: current_term: 1 leader_uuid: "97175f85e6784f6bb1c4f433104a3802" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "744ee3020a694d7fad3c0ee1c7e50a26" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 37593 } } peers { permanent_uuid: "97175f85e6784f6bb1c4f433104a3802" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 46249 } } peers { permanent_uuid: "fe48f25970b145629c3e36b146488cb0" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 34979 } } }
I20260929 20:22:31.692592 19270 tablet_replica.cc:442] T f3191d83a7e344bd8487b0e21a516e9f P fe48f25970b145629c3e36b146488cb0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 20:22:31.694231 19283 tablet_replica.cc:439] T f3191d83a7e344bd8487b0e21a516e9f P 97175f85e6784f6bb1c4f433104a3802: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "97175f85e6784f6bb1c4f433104a3802" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "744ee3020a694d7fad3c0ee1c7e50a26" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 37593 } } peers { permanent_uuid: "97175f85e6784f6bb1c4f433104a3802" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 46249 } } peers { permanent_uuid: "fe48f25970b145629c3e36b146488cb0" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 34979 } } }
I20260929 20:22:31.694559 19283 tablet_replica.cc:442] T f3191d83a7e344bd8487b0e21a516e9f P 97175f85e6784f6bb1c4f433104a3802: This TxnStatusTablet replica's current role is: LEADER
I20260929 20:22:31.699350 19270 tablet_replica.cc:439] T f3191d83a7e344bd8487b0e21a516e9f P fe48f25970b145629c3e36b146488cb0: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "97175f85e6784f6bb1c4f433104a3802" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "744ee3020a694d7fad3c0ee1c7e50a26" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 37593 } } peers { permanent_uuid: "97175f85e6784f6bb1c4f433104a3802" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 46249 } } peers { permanent_uuid: "fe48f25970b145629c3e36b146488cb0" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 34979 } } }
I20260929 20:22:31.699558 19270 tablet_replica.cc:442] T f3191d83a7e344bd8487b0e21a516e9f P fe48f25970b145629c3e36b146488cb0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 20:22:31.713701 19310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:31.707303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.53.66:34063 (local address 127.17.53.66:46249)
0929 20:22:31.707500 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0929 20:22:31.707508 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 20:22:31.708135 (+ 627us) server_negotiation.cc:407] Connection header received
0929 20:22:31.708277 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:22:31.708283 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:22:31.708382 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:22:31.708485 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 20:22:31.708971 (+ 486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:31.710152 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:31.711689 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:31.712380 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:31.712500 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:22:31.712888 (+ 388us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 20:22:31.713014 (+ 126us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 20:22:31.713116 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0929 20:22:31.713307 (+ 191us) server_negotiation.cc:299] Negotiation successful
0929 20:22:31.713385 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 20:22:31.713707 19318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:31.707300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.66:46249 (local address 127.17.53.66:34063)
0929 20:22:31.708041 (+ 741us) negotiation.cc:107] Waiting for socket to connect
0929 20:22:31.708059 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 20:22:31.708169 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:22:31.708495 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:22:31.708498 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:22:31.708510 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 20:22:31.708820 (+ 310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:22:31.708828 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:22:31.710277 (+ 1449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:22:31.710280 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:22:31.711557 (+ 1277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:22:31.711568 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:22:31.712554 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:22:31.712559 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:22:31.712676 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:22:31.712729 (+ 53us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 20:22:31.713153 (+ 424us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 20:22:31.713185 (+ 32us) client_negotiation.cc:769] Sending connection context
0929 20:22:31.713255 (+ 70us) client_negotiation.cc:240] Negotiation successful
0929 20:22:31.713366 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":596,"thread_start_us":380,"threads_started":1}
I20260929 20:22:31.778313 18752 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49096:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 20:22:31.778560 18752 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49096:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 20:22:31.781575 18752 catalog_manager.cc:5782] T 00000000000000000000000000000000 P edc009a06883495fbef928dda0915f0d: Sending DeleteTablet for 3 replicas of tablet bd1284f0dede47d98c087b6e948637af
I20260929 20:22:31.782449 19174 tablet_service.cc:1514] Processing DeleteTablet for tablet bd1284f0dede47d98c087b6e948637af with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:22:31 UTC) from {username='kudu', principal='kudu/127.17.53.126@KRBTEST.COM'} at 127.0.0.1:49554
I20260929 20:22:31.783049 18893 tablet_service.cc:1514] Processing DeleteTablet for tablet bd1284f0dede47d98c087b6e948637af with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:22:31 UTC) from {username='kudu', principal='kudu/127.17.53.126@KRBTEST.COM'} at 127.0.0.1:37860
I20260929 20:22:31.785216 19034 tablet_service.cc:1514] Processing DeleteTablet for tablet bd1284f0dede47d98c087b6e948637af with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:22:31 UTC) from {username='kudu', principal='kudu/127.17.53.126@KRBTEST.COM'} at 127.0.0.1:48808
I20260929 20:22:31.794843 19320 tablet_replica.cc:330] T bd1284f0dede47d98c087b6e948637af P fe48f25970b145629c3e36b146488cb0: stopping tablet replica
I20260929 20:22:31.795127 19320 raft_consensus.cc:2229] T bd1284f0dede47d98c087b6e948637af P fe48f25970b145629c3e36b146488cb0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 20:22:31.795383 19320 raft_consensus.cc:2258] T bd1284f0dede47d98c087b6e948637af P fe48f25970b145629c3e36b146488cb0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:22:31.796550 19320 ts_tablet_manager.cc:1905] T bd1284f0dede47d98c087b6e948637af P fe48f25970b145629c3e36b146488cb0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:22:31.799142 19321 tablet_replica.cc:330] T bd1284f0dede47d98c087b6e948637af P 744ee3020a694d7fad3c0ee1c7e50a26: stopping tablet replica
I20260929 20:22:31.799352 19321 raft_consensus.cc:2229] T bd1284f0dede47d98c087b6e948637af P 744ee3020a694d7fad3c0ee1c7e50a26 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 20:22:31.799603 19321 raft_consensus.cc:2258] T bd1284f0dede47d98c087b6e948637af P 744ee3020a694d7fad3c0ee1c7e50a26 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:22:31.800865 19321 ts_tablet_manager.cc:1905] T bd1284f0dede47d98c087b6e948637af P 744ee3020a694d7fad3c0ee1c7e50a26: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:22:31.802062 19323 tablet_replica.cc:330] T bd1284f0dede47d98c087b6e948637af P 97175f85e6784f6bb1c4f433104a3802: stopping tablet replica
I20260929 20:22:31.802309 19323 raft_consensus.cc:2229] T bd1284f0dede47d98c087b6e948637af P 97175f85e6784f6bb1c4f433104a3802 [term 1 LEADER]: Raft consensus shutting down.
I20260929 20:22:31.802707 19323 raft_consensus.cc:2258] T bd1284f0dede47d98c087b6e948637af P 97175f85e6784f6bb1c4f433104a3802 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:22:31.804443 19320 ts_tablet_manager.cc:1918] T bd1284f0dede47d98c087b6e948637af P fe48f25970b145629c3e36b146488cb0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 20:22:31.804567 19320 log.cc:1196] T bd1284f0dede47d98c087b6e948637af P fe48f25970b145629c3e36b146488cb0: Deleting WAL directory at /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790713339161492-17621-0/minicluster-data/ts-2/wal/wals/bd1284f0dede47d98c087b6e948637af
I20260929 20:22:31.804992 19320 ts_tablet_manager.cc:1939] T bd1284f0dede47d98c087b6e948637af P fe48f25970b145629c3e36b146488cb0: Deleting consensus metadata
I20260929 20:22:31.806057 18739 catalog_manager.cc:4841] TS fe48f25970b145629c3e36b146488cb0 (127.17.53.67:34979): tablet bd1284f0dede47d98c087b6e948637af (table test-table [id=fae0c4a2f57641c5bd185c7d460f5180]) successfully deleted
I20260929 20:22:31.813227 19321 ts_tablet_manager.cc:1918] T bd1284f0dede47d98c087b6e948637af P 744ee3020a694d7fad3c0ee1c7e50a26: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 20:22:31.813385 19321 log.cc:1196] T bd1284f0dede47d98c087b6e948637af P 744ee3020a694d7fad3c0ee1c7e50a26: Deleting WAL directory at /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790713339161492-17621-0/minicluster-data/ts-0/wal/wals/bd1284f0dede47d98c087b6e948637af
I20260929 20:22:31.813828 19321 ts_tablet_manager.cc:1939] T bd1284f0dede47d98c087b6e948637af P 744ee3020a694d7fad3c0ee1c7e50a26: Deleting consensus metadata
I20260929 20:22:31.815209 18737 catalog_manager.cc:4841] TS 744ee3020a694d7fad3c0ee1c7e50a26 (127.17.53.65:37593): tablet bd1284f0dede47d98c087b6e948637af (table test-table [id=fae0c4a2f57641c5bd185c7d460f5180]) successfully deleted
I20260929 20:22:31.816215 19323 ts_tablet_manager.cc:1905] T bd1284f0dede47d98c087b6e948637af P 97175f85e6784f6bb1c4f433104a3802: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:22:31.823851 19323 ts_tablet_manager.cc:1918] T bd1284f0dede47d98c087b6e948637af P 97175f85e6784f6bb1c4f433104a3802: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 20:22:31.824023 19323 log.cc:1196] T bd1284f0dede47d98c087b6e948637af P 97175f85e6784f6bb1c4f433104a3802: Deleting WAL directory at /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790713339161492-17621-0/minicluster-data/ts-1/wal/wals/bd1284f0dede47d98c087b6e948637af
I20260929 20:22:31.824457 19323 ts_tablet_manager.cc:1939] T bd1284f0dede47d98c087b6e948637af P 97175f85e6784f6bb1c4f433104a3802: Deleting consensus metadata
I20260929 20:22:31.825493 18740 catalog_manager.cc:4841] TS 97175f85e6784f6bb1c4f433104a3802 (127.17.53.66:46249): tablet bd1284f0dede47d98c087b6e948637af (table test-table [id=fae0c4a2f57641c5bd185c7d460f5180]) successfully deleted
Sep 29 20:22:31 dist-test-slave-zbnc krb5kdc[18708](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713350, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.17.53.65@KRBTEST.COM
I20260929 20:22:31.835866 19324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:31.799333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37876 (local address 127.17.53.65:37593)
0929 20:22:31.807304 (+ 7971us) server_negotiation.cc:206] Beginning negotiation
0929 20:22:31.807313 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 20:22:31.815364 (+ 8051us) server_negotiation.cc:407] Connection header received
0929 20:22:31.815581 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:22:31.815586 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:22:31.815659 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:22:31.815763 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:22:31.818215 (+ 2452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:31.819435 (+ 1220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:31.821415 (+ 1980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:31.822113 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:31.822212 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:22:31.828373 (+ 6161us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:22:31.828410 (+ 37us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:22:31.828431 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:22:31.828479 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:22:31.832616 (+ 4137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:22:31.833480 (+ 864us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:22:31.833491 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:22:31.833495 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:22:31.833680 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:22:31.834098 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:22:31.834103 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:22:31.834106 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:22:31.834558 (+ 452us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:22:31.834909 (+ 351us) server_negotiation.cc:1035] Waiting for connection context
0929 20:22:31.835197 (+ 288us) server_negotiation.cc:299] Negotiation successful
0929 20:22:31.835438 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7761,"thread_start_us":7656,"threads_started":1}
W20260929 20:22:31.836790 18959 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37876
I20260929 20:22:31.855065 19328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:31.840207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49112 (local address 127.17.53.126:43983)
0929 20:22:31.841359 (+ 1152us) server_negotiation.cc:206] Beginning negotiation
0929 20:22:31.841363 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:22:31.841378 (+ 15us) server_negotiation.cc:407] Connection header received
0929 20:22:31.841427 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:22:31.841430 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:22:31.841487 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:22:31.841572 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:22:31.843401 (+ 1829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:31.844224 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:31.846087 (+ 1863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:31.846624 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:31.846725 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:22:31.849690 (+ 2965us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:22:31.849695 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:22:31.849698 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:22:31.849731 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:22:31.852495 (+ 2764us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:22:31.853316 (+ 821us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:22:31.853319 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:22:31.853321 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:22:31.853429 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:22:31.853783 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:22:31.853786 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:22:31.853788 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:22:31.854040 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:22:31.854208 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0929 20:22:31.854562 (+ 354us) server_negotiation.cc:299] Negotiation successful
0929 20:22:31.854714 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1030,"thread_start_us":900,"threads_started":1}
W20260929 20:22:31.855666 18752 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49112
I20260929 20:22:31.857668 17621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu pid 18826
I20260929 20:22:31.874284 19324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:31.859049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37882 (local address 127.17.53.65:37593)
0929 20:22:31.859622 (+ 573us) server_negotiation.cc:206] Beginning negotiation
0929 20:22:31.859628 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:22:31.859645 (+ 17us) server_negotiation.cc:407] Connection header received
0929 20:22:31.859702 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:22:31.859707 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:22:31.859784 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:22:31.859884 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:22:31.860933 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:31.862068 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:31.863900 (+ 1832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:31.864553 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:31.864728 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:22:31.867575 (+ 2847us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:22:31.867581 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:22:31.867586 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:22:31.867628 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:22:31.871235 (+ 3607us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:22:31.872074 (+ 839us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:22:31.872080 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:22:31.872084 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:22:31.872229 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:22:31.872623 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:22:31.872628 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:22:31.872632 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:22:31.872960 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:22:31.873230 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0929 20:22:31.873528 (+ 298us) server_negotiation.cc:299] Negotiation successful
0929 20:22:31.873759 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":445}
W20260929 20:22:31.874846 18959 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37882
W20260929 20:22:31.875478 17621 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu pid 18826: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 20:22:31.875654 17621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu with pid 18826
I20260929 20:22:31.898190 17621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu pid 18968
Sep 29 20:22:31 dist-test-slave-zbnc krb5kdc[18708](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713350, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.17.53.66@KRBTEST.COM
I20260929 20:22:31.917055 19310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:31.899054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48836 (local address 127.17.53.66:46249)
0929 20:22:31.899300 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0929 20:22:31.899307 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:22:31.899328 (+ 21us) server_negotiation.cc:407] Connection header received
0929 20:22:31.899463 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:22:31.899469 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:22:31.899564 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:22:31.899680 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:22:31.901212 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:31.902423 (+ 1211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:31.904323 (+ 1900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:31.905071 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:31.905211 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:22:31.910371 (+ 5160us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:22:31.910389 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:22:31.910409 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:22:31.910451 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:22:31.914127 (+ 3676us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:22:31.914964 (+ 837us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:22:31.914974 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:22:31.914977 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:22:31.915114 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:22:31.915517 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:22:31.915520 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:22:31.915522 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:22:31.915875 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:22:31.916105 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0929 20:22:31.916516 (+ 411us) server_negotiation.cc:299] Negotiation successful
0929 20:22:31.916692 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109}
W20260929 20:22:31.917805 19100 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48836
W20260929 20:22:31.918519 17621 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu pid 18968: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 20:22:31.918694 17621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu with pid 18968
I20260929 20:22:31.941502 17621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu pid 19109
Sep 29 20:22:31 dist-test-slave-zbnc krb5kdc[18708](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713350, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.17.53.67@KRBTEST.COM
I20260929 20:22:31.956683 19330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:31.942361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49570 (local address 127.17.53.67:34979)
0929 20:22:31.942818 (+ 457us) server_negotiation.cc:206] Beginning negotiation
0929 20:22:31.942824 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:22:31.942842 (+ 18us) server_negotiation.cc:407] Connection header received
0929 20:22:31.942970 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:22:31.942973 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:22:31.943041 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:22:31.943130 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:22:31.944350 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:31.945161 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:31.946342 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:31.946978 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:31.947079 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:22:31.951200 (+ 4121us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:22:31.951215 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:22:31.951231 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:22:31.951261 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:22:31.954036 (+ 2775us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:22:31.954654 (+ 618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:22:31.954662 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:22:31.954664 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:22:31.954820 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:22:31.955333 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:22:31.955336 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:22:31.955338 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:22:31.955668 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:22:31.955868 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0929 20:22:31.956142 (+ 274us) server_negotiation.cc:299] Negotiation successful
0929 20:22:31.956301 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":347,"thread_start_us":233,"threads_started":1}
W20260929 20:22:31.957377 19240 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49570
W20260929 20:22:31.958007 17621 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu pid 19109: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 20:22:31.958175 17621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu with pid 19109
I20260929 20:22:31.975407 17621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu pid 18721
I20260929 20:22:31.976522 18794 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:22:32.091363 17621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu with pid 18721
2026-09-29T20:22:32Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (4667 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790713339161492-17621-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 20:22:32 dist-test-slave-zbnc krb5kdc[19334](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 20:22:32 dist-test-slave-zbnc krb5kdc[19334](info): set up 2 sockets
Sep 29 20:22:32 dist-test-slave-zbnc krb5kdc[19334](info): commencing operation
krb5kdc: starting...
W20260929 20:22:33.200425 17621 mini_kdc.cc:121] Time spent starting KDC: real 1.065s user 0.003s sys 0.012s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790713339161492-17621-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790713339161492-17621-0/krb5kdc/test-user.keytab.
Sep 29 20:22:33 dist-test-slave-zbnc krb5kdc[19334](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713353, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T20:22:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T20:22:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.17.53.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.53.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.53.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.126.keytab.
Entry for principal kudu/127.17.53.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.126.keytab.
I20260929 20:22:33.392184 17621 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
/tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790713339161492-17621-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790713339161492-17621-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790713339161492-17621-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790713339161492-17621-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.53.126:42509
--webserver_interface=127.17.53.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.126.keytab
--principal=kudu/127.17.53.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.53.84:39917
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.53.126:42509
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790713339161492-17621-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790713339161492-17621-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790713339161492-17621-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:22:33.737941 19348 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:22:33.738307 19348 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:22:33.738488 19348 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:22:33.746775 19348 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:22:33.746978 19348 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260929 20:22:33.747076 19348 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:22:33.747218 19348 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:22:33.747316 19348 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:22:33.747426 19348 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:22:33.758387 19348 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.53.84:39917
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790713339161492-17621-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790713339161492-17621-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.53.126:42509
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.126.keytab
--principal=kudu/127.17.53.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.53.126:42509
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790713339161492-17621-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.53.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790713339161492-17621-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:22:33.759821 19348 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:22:33.761260 19348 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:22:33.774225 19353 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:22:33.774189 19356 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:22:33.774314 19354 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:22:33.775027 19348 server_base.cc:1034] running on GCE node
I20260929 20:22:33.775864 19348 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:22:33.777287 19348 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:22:33.778548 19348 hybrid_clock.cc:648] HybridClock initialized: now 1790713353778490 us; error 48 us; skew 500 ppm
I20260929 20:22:33.779047 19348 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:22:33 dist-test-slave-zbnc krb5kdc[19334](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713353, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.53.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:22:33.782313 19348 init.cc:377] Logged in from keytab as kudu/127.17.53.126@KRBTEST.COM (short username kudu)
I20260929 20:22:33.783881 19348 webserver.cc:460] Webserver started at http://127.17.53.126:40337/ using document root <none> and password file <none>
I20260929 20:22:33.784600 19348 fs_manager.cc:359] Metadata directory not provided
I20260929 20:22:33.784688 19348 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:22:33.784970 19348 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:22:33.788098 19348 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790713339161492-17621-0/minicluster-data/master-0/data/instance:
uuid: "dd8eef272ec6403e935a5cbc0ef50dce"
format_stamp: "Formatted at 2026-09-29 20:22:33 on dist-test-slave-zbnc"
server_key: "c35aa8eac53df9ff3b190bb69daf59ca"
server_key_iv: "e1278414e1fab6ab1f9fc6397863e90b"
server_key_version: "encryptionkey@0"
I20260929 20:22:33.788998 19348 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790713339161492-17621-0/minicluster-data/master-0/wal/instance:
uuid: "dd8eef272ec6403e935a5cbc0ef50dce"
format_stamp: "Formatted at 2026-09-29 20:22:33 on dist-test-slave-zbnc"
server_key: "c35aa8eac53df9ff3b190bb69daf59ca"
server_key_iv: "e1278414e1fab6ab1f9fc6397863e90b"
server_key_version: "encryptionkey@0"
I20260929 20:22:33.794374 19348 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260929 20:22:33.798132 19363 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:22:33.799669 19348 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 20:22:33.799855 19348 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790713339161492-17621-0/minicluster-data/master-0/data,/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790713339161492-17621-0/minicluster-data/master-0/wal
uuid: "dd8eef272ec6403e935a5cbc0ef50dce"
format_stamp: "Formatted at 2026-09-29 20:22:33 on dist-test-slave-zbnc"
server_key: "c35aa8eac53df9ff3b190bb69daf59ca"
server_key_iv: "e1278414e1fab6ab1f9fc6397863e90b"
server_key_version: "encryptionkey@0"
I20260929 20:22:33.800094 19348 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790713339161492-17621-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790713339161492-17621-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790713339161492-17621-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:22:33.825227 19348 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:22:33.829753 19348 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:22:33.830060 19348 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:22:33.854264 19348 rpc_server.cc:307] RPC server started. Bound to: 127.17.53.126:42509
I20260929 20:22:33.854298 19424 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.53.126:42509 every 8 connection(s)
I20260929 20:22:33.855880 19348 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790713339161492-17621-0/minicluster-data/master-0/data/info.pb
I20260929 20:22:33.859715 19425 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:22:33.860224 17621 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu as pid 19348
I20260929 20:22:33.860442 17621 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790713339161492-17621-0/minicluster-data/master-0/wal/instance
I20260929 20:22:33.861128 17621 external_mini_cluster.cc:1428] Setting key e97082c0ef17d3d51133219cb78573e0
I20260929 20:22:33.872676 19425 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dd8eef272ec6403e935a5cbc0ef50dce: Bootstrap starting.
Sep 29 20:22:33 dist-test-slave-zbnc krb5kdc[19334](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713353, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.53.126@KRBTEST.COM
I20260929 20:22:33.876767 19425 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dd8eef272ec6403e935a5cbc0ef50dce: Neither blocks nor log segments found. Creating new log.
I20260929 20:22:33.877808 19425 log.cc:824] T 00000000000000000000000000000000 P dd8eef272ec6403e935a5cbc0ef50dce: Log is configured to *not* fsync() on all Append() calls
I20260929 20:22:33.880661 19425 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dd8eef272ec6403e935a5cbc0ef50dce: No bootstrap required, opened a new log
I20260929 20:22:33.885036 19428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:33.863350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60610 (local address 127.17.53.126:42509)
0929 20:22:33.864388 (+ 1038us) server_negotiation.cc:206] Beginning negotiation
0929 20:22:33.864406 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0929 20:22:33.864715 (+ 309us) server_negotiation.cc:407] Connection header received
0929 20:22:33.866640 (+ 1925us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:22:33.866663 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:22:33.867157 (+ 494us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:22:33.867575 (+ 418us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:22:33.869148 (+ 1573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:33.870639 (+ 1491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:33.872077 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:33.872801 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:33.872997 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:22:33.877708 (+ 4711us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:22:33.877716 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:22:33.877731 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:22:33.877782 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:22:33.880996 (+ 3214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:22:33.881738 (+ 742us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:22:33.881750 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:22:33.881756 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:22:33.881923 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:22:33.882305 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:22:33.882309 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:22:33.882311 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:22:33.882811 (+ 500us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:22:33.883213 (+ 402us) server_negotiation.cc:1035] Waiting for connection context
0929 20:22:33.883677 (+ 464us) server_negotiation.cc:299] Negotiation successful
0929 20:22:33.884084 (+ 407us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":730,"thread_start_us":473,"threads_started":1}
I20260929 20:22:33.888764 19425 raft_consensus.cc:348] T 00000000000000000000000000000000 P dd8eef272ec6403e935a5cbc0ef50dce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd8eef272ec6403e935a5cbc0ef50dce" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 42509 } }
I20260929 20:22:33.889181 19425 raft_consensus.cc:374] T 00000000000000000000000000000000 P dd8eef272ec6403e935a5cbc0ef50dce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:22:33.889287 19425 raft_consensus.cc:729] T 00000000000000000000000000000000 P dd8eef272ec6403e935a5cbc0ef50dce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd8eef272ec6403e935a5cbc0ef50dce, State: Initialized, Role: FOLLOWER
I20260929 20:22:33.889842 19425 consensus_queue.cc:260] T 00000000000000000000000000000000 P dd8eef272ec6403e935a5cbc0ef50dce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd8eef272ec6403e935a5cbc0ef50dce" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 42509 } }
I20260929 20:22:33.890085 19425 raft_consensus.cc:388] T 00000000000000000000000000000000 P dd8eef272ec6403e935a5cbc0ef50dce [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 20:22:33.890237 19425 raft_consensus.cc:482] T 00000000000000000000000000000000 P dd8eef272ec6403e935a5cbc0ef50dce [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 20:22:33.890409 19425 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dd8eef272ec6403e935a5cbc0ef50dce [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:22:33.892637 19425 raft_consensus.cc:504] T 00000000000000000000000000000000 P dd8eef272ec6403e935a5cbc0ef50dce [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd8eef272ec6403e935a5cbc0ef50dce" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 42509 } }
I20260929 20:22:33.893118 19425 leader_election.cc:302] T 00000000000000000000000000000000 P dd8eef272ec6403e935a5cbc0ef50dce [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dd8eef272ec6403e935a5cbc0ef50dce; no voters:
I20260929 20:22:33.893677 19425 leader_election.cc:288] T 00000000000000000000000000000000 P dd8eef272ec6403e935a5cbc0ef50dce [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 20:22:33.893805 19430 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dd8eef272ec6403e935a5cbc0ef50dce [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:22:33.895538 19430 raft_consensus.cc:686] T 00000000000000000000000000000000 P dd8eef272ec6403e935a5cbc0ef50dce [term 1 LEADER]: Becoming Leader. State: Replica: dd8eef272ec6403e935a5cbc0ef50dce, State: Running, Role: LEADER
I20260929 20:22:33.895885 19425 sys_catalog.cc:564] T 00000000000000000000000000000000 P dd8eef272ec6403e935a5cbc0ef50dce [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:22:33.896138 19430 consensus_queue.cc:237] T 00000000000000000000000000000000 P dd8eef272ec6403e935a5cbc0ef50dce [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd8eef272ec6403e935a5cbc0ef50dce" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 42509 } }
I20260929 20:22:33.899017 19431 sys_catalog.cc:455] T 00000000000000000000000000000000 P dd8eef272ec6403e935a5cbc0ef50dce [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dd8eef272ec6403e935a5cbc0ef50dce" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd8eef272ec6403e935a5cbc0ef50dce" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 42509 } } }
I20260929 20:22:33.898993 19432 sys_catalog.cc:455] T 00000000000000000000000000000000 P dd8eef272ec6403e935a5cbc0ef50dce [sys.catalog]: SysCatalogTable state changed. Reason: New leader dd8eef272ec6403e935a5cbc0ef50dce. Latest consensus state: current_term: 1 leader_uuid: "dd8eef272ec6403e935a5cbc0ef50dce" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dd8eef272ec6403e935a5cbc0ef50dce" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 42509 } } }
I20260929 20:22:33.899435 19431 sys_catalog.cc:458] T 00000000000000000000000000000000 P dd8eef272ec6403e935a5cbc0ef50dce [sys.catalog]: This master's current role is: LEADER
I20260929 20:22:33.899435 19432 sys_catalog.cc:458] T 00000000000000000000000000000000 P dd8eef272ec6403e935a5cbc0ef50dce [sys.catalog]: This master's current role is: LEADER
I20260929 20:22:33.900290 19437 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:22:33.905714 19437 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:22:33.914446 19437 catalog_manager.cc:1347] Generated new cluster ID: 431e359533c1470984b697167dfcfd03
I20260929 20:22:33.914569 19437 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:22:33.939040 19437 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 20:22:33.940454 19437 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:22:33.954926 19437 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dd8eef272ec6403e935a5cbc0ef50dce: Generated new TSK 0
I20260929 20:22:33.955847 19437 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.17.53.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.53.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.53.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.65.keytab.
Entry for principal kudu/127.17.53.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.65.keytab.
I20260929 20:22:34.011195 17621 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
/tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790713339161492-17621-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790713339161492-17621-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790713339161492-17621-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790713339161492-17621-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.53.65:0
--local_ip_for_outbound_sockets=127.17.53.65
--webserver_interface=127.17.53.65
--webserver_port=0
--tserver_master_addrs=127.17.53.126:42509
--keytab_file=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.65.keytab
--principal=kudu/127.17.53.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.53.84:39917
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790713339161492-17621-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790713339161492-17621-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790713339161492-17621-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:22:34.341313 19451 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:22:34.341614 19451 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:22:34.341796 19451 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:22:34.349277 19451 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:22:34.349392 19451 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:22:34.349586 19451 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260929 20:22:34.349663 19451 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 20:22:34.349794 19451 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.53.65
I20260929 20:22:34.359862 19451 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.53.84:39917
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790713339161492-17621-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790713339161492-17621-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.65.keytab
--principal=kudu/127.17.53.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.53.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790713339161492-17621-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.17.53.65
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.17.53.126:42509
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.53.65
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790713339161492-17621-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:22:34.361213 19451 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:22:34.362530 19451 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:22:34.375969 19457 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:22:34.376034 19456 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:22:34.375998 19459 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:22:34.376957 19451 server_base.cc:1034] running on GCE node
I20260929 20:22:34.377521 19451 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:22:34.378449 19451 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:22:34.379650 19451 hybrid_clock.cc:648] HybridClock initialized: now 1790713354379617 us; error 32 us; skew 500 ppm
I20260929 20:22:34.379940 19451 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:22:34 dist-test-slave-zbnc krb5kdc[19334](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713354, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.53.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:22:34.383080 19451 init.cc:377] Logged in from keytab as kudu/127.17.53.65@KRBTEST.COM (short username kudu)
I20260929 20:22:34.384572 19451 webserver.cc:460] Webserver started at http://127.17.53.65:36821/ using document root <none> and password file <none>
I20260929 20:22:34.385272 19451 fs_manager.cc:359] Metadata directory not provided
I20260929 20:22:34.385452 19451 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:22:34.385793 19451 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:22:34.389017 19451 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790713339161492-17621-0/minicluster-data/ts-0/data/instance:
uuid: "ba9328e853e140d6b28c75c85f061c69"
format_stamp: "Formatted at 2026-09-29 20:22:34 on dist-test-slave-zbnc"
server_key: "0166267052f0282ed7ff8cfe852b85a0"
server_key_iv: "3322e1d5effc92b4dc999f49e67336a7"
server_key_version: "encryptionkey@0"
I20260929 20:22:34.389815 19451 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790713339161492-17621-0/minicluster-data/ts-0/wal/instance:
uuid: "ba9328e853e140d6b28c75c85f061c69"
format_stamp: "Formatted at 2026-09-29 20:22:34 on dist-test-slave-zbnc"
server_key: "0166267052f0282ed7ff8cfe852b85a0"
server_key_iv: "3322e1d5effc92b4dc999f49e67336a7"
server_key_version: "encryptionkey@0"
I20260929 20:22:34.395009 19451 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260929 20:22:34.398552 19466 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:22:34.399917 19451 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 20:22:34.400092 19451 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790713339161492-17621-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790713339161492-17621-0/minicluster-data/ts-0/wal
uuid: "ba9328e853e140d6b28c75c85f061c69"
format_stamp: "Formatted at 2026-09-29 20:22:34 on dist-test-slave-zbnc"
server_key: "0166267052f0282ed7ff8cfe852b85a0"
server_key_iv: "3322e1d5effc92b4dc999f49e67336a7"
server_key_version: "encryptionkey@0"
I20260929 20:22:34.400291 19451 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790713339161492-17621-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790713339161492-17621-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790713339161492-17621-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:22:34.414988 19451 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:22:34.419118 19451 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:22:34.419418 19451 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:22:34.422029 19451 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:22:34.422180 19451 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:22:34.422317 19451 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:22:34.422413 19451 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 20:22:34 dist-test-slave-zbnc krb5kdc[19334](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713354, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.53.65@KRBTEST.COM for kudu/127.17.53.126@KRBTEST.COM
I20260929 20:22:34.449676 19476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:34.423452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:42509 (local address 127.17.53.65:41437)
0929 20:22:34.424351 (+ 899us) negotiation.cc:107] Waiting for socket to connect
0929 20:22:34.424408 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0929 20:22:34.426676 (+ 2268us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:22:34.427583 (+ 907us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:22:34.427599 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:22:34.428566 (+ 967us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:22:34.429681 (+ 1115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:22:34.429712 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:22:34.431338 (+ 1626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:22:34.431350 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:22:34.434034 (+ 2684us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:22:34.434056 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:22:34.435308 (+ 1252us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:22:34.435313 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:22:34.435592 (+ 279us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:22:34.436538 (+ 946us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:22:34.436588 (+ 50us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:22:34.440160 (+ 3572us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:22:34.444658 (+ 4498us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:22:34.444671 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:22:34.444697 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:22:34.445397 (+ 700us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:22:34.445839 (+ 442us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:22:34.445849 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:22:34.445851 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:22:34.446185 (+ 334us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:22:34.446851 (+ 666us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:22:34.446883 (+ 32us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:22:34.447473 (+ 590us) client_negotiation.cc:769] Sending connection context
0929 20:22:34.447829 (+ 356us) client_negotiation.cc:240] Negotiation successful
0929 20:22:34.448374 (+ 545us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":630,"thread_start_us":394,"threads_started":1}
I20260929 20:22:34.450811 19475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:34.422883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.53.65:41437 (local address 127.17.53.126:42509)
0929 20:22:34.425171 (+ 2288us) server_negotiation.cc:206] Beginning negotiation
0929 20:22:34.425178 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:22:34.425233 (+ 55us) server_negotiation.cc:407] Connection header received
0929 20:22:34.427032 (+ 1799us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:22:34.427037 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:22:34.427125 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:22:34.427257 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:22:34.429928 (+ 2671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:34.431082 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:34.434258 (+ 3176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:34.435077 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:34.435225 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:22:34.440460 (+ 5235us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:22:34.440468 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:22:34.440472 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:22:34.440521 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:22:34.444481 (+ 3960us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:22:34.445584 (+ 1103us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:22:34.445588 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:22:34.445590 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:22:34.445706 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:22:34.446357 (+ 651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:22:34.446365 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:22:34.446369 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:22:34.446694 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:22:34.446880 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0929 20:22:34.450374 (+ 3494us) server_negotiation.cc:299] Negotiation successful
0929 20:22:34.450570 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2148,"thread_start_us":2045,"threads_started":1}
I20260929 20:22:34.471810 19551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:34.457038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:42509 (local address 127.17.53.65:58191)
0929 20:22:34.457772 (+ 734us) negotiation.cc:107] Waiting for socket to connect
0929 20:22:34.457796 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0929 20:22:34.457972 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:22:34.458363 (+ 391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:22:34.458367 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:22:34.459124 (+ 757us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:22:34.459705 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:22:34.459721 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:22:34.461064 (+ 1343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:22:34.461069 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:22:34.462706 (+ 1637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:22:34.462718 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:22:34.463769 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:22:34.463776 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:22:34.463908 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:22:34.464578 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:22:34.464595 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:22:34.465290 (+ 695us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:22:34.468642 (+ 3352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:22:34.468651 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:22:34.468655 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:22:34.469227 (+ 572us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:22:34.469743 (+ 516us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:22:34.469748 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:22:34.469751 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:22:34.469959 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:22:34.470708 (+ 749us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:22:34.470712 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:22:34.470929 (+ 217us) client_negotiation.cc:769] Sending connection context
0929 20:22:34.471151 (+ 222us) client_negotiation.cc:240] Negotiation successful
0929 20:22:34.471476 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":525,"thread_start_us":364,"threads_started":1}
I20260929 20:22:34.472067 19475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:34.456888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.53.65:58191 (local address 127.17.53.126:42509)
0929 20:22:34.457090 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0929 20:22:34.457099 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 20:22:34.457908 (+ 809us) server_negotiation.cc:407] Connection header received
0929 20:22:34.458125 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:22:34.458129 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:22:34.458190 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:22:34.458328 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:22:34.459864 (+ 1536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:34.460931 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:34.462902 (+ 1971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:34.463319 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:34.463471 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:22:34.465474 (+ 2003us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:22:34.465480 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:22:34.465486 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:22:34.465523 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:22:34.468471 (+ 2948us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:22:34.469427 (+ 956us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:22:34.469435 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:22:34.469441 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:22:34.469618 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:22:34.470118 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:22:34.470124 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:22:34.470130 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:22:34.470570 (+ 440us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:22:34.470748 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0929 20:22:34.471284 (+ 536us) server_negotiation.cc:299] Negotiation successful
0929 20:22:34.471597 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260929 20:22:34.475449 19451 rpc_server.cc:307] RPC server started. Bound to: 127.17.53.65:43187
I20260929 20:22:34.475533 19587 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.53.65:43187 every 8 connection(s)
I20260929 20:22:34.476698 19451 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790713339161492-17621-0/minicluster-data/ts-0/data/info.pb
I20260929 20:22:34.478559 19588 heartbeater.cc:347] Connected to a master server at 127.17.53.126:42509
I20260929 20:22:34.478793 19588 heartbeater.cc:464] Registering TS with master...
I20260929 20:22:34.479418 17621 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu as pid 19451
I20260929 20:22:34.479521 19588 heartbeater.cc:511] Master 127.17.53.126:42509 requested a full tablet report, sending...
I20260929 20:22:34.479604 17621 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790713339161492-17621-0/minicluster-data/ts-0/wal/instance
I20260929 20:22:34.480232 17621 external_mini_cluster.cc:1428] Setting key 2b4c0c5a78da0204fdd5a6d4af01af8a
I20260929 20:22:34.481549 19380 ts_manager.cc:194] Registered new tserver with Master: ba9328e853e140d6b28c75c85f061c69 (127.17.53.65:43187)
I20260929 20:22:34.483127 19380 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.53.65@KRBTEST.COM'} at 127.17.53.65:41437
WARNING: no policy specified for kudu/127.17.53.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.53.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.53.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.66.keytab.
Entry for principal kudu/127.17.53.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.66.keytab.
I20260929 20:22:34.522733 17621 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
/tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790713339161492-17621-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790713339161492-17621-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790713339161492-17621-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790713339161492-17621-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.53.66:0
--local_ip_for_outbound_sockets=127.17.53.66
--webserver_interface=127.17.53.66
--webserver_port=0
--tserver_master_addrs=127.17.53.126:42509
--keytab_file=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.66.keytab
--principal=kudu/127.17.53.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.53.84:39917
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790713339161492-17621-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790713339161492-17621-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790713339161492-17621-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:22:34.860069 19592 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:22:34.860427 19592 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:22:34.860635 19592 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:22:34.868947 19592 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:22:34.869108 19592 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:22:34.869292 19592 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260929 20:22:34.869396 19592 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 20:22:34.869550 19592 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.53.66
I20260929 20:22:34.880653 19592 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.53.84:39917
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790713339161492-17621-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790713339161492-17621-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.66.keytab
--principal=kudu/127.17.53.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.53.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790713339161492-17621-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.17.53.66
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.17.53.126:42509
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.53.66
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790713339161492-17621-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:22:34.882079 19592 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:22:34.883621 19592 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:22:34.897974 19600 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:22:34.898003 19598 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:22:34.898072 19597 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:22:34.899367 19592 server_base.cc:1034] running on GCE node
I20260929 20:22:34.900121 19592 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:22:34.901177 19592 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:22:34.902431 19592 hybrid_clock.cc:648] HybridClock initialized: now 1790713354902377 us; error 54 us; skew 500 ppm
I20260929 20:22:34.902771 19592 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:22:34 dist-test-slave-zbnc krb5kdc[19334](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713354, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.53.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:22:34.910840 19592 init.cc:377] Logged in from keytab as kudu/127.17.53.66@KRBTEST.COM (short username kudu)
I20260929 20:22:34.912477 19592 webserver.cc:460] Webserver started at http://127.17.53.66:46731/ using document root <none> and password file <none>
I20260929 20:22:34.913213 19592 fs_manager.cc:359] Metadata directory not provided
I20260929 20:22:34.913384 19592 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:22:34.913740 19592 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:22:34.916857 19592 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790713339161492-17621-0/minicluster-data/ts-1/data/instance:
uuid: "fc610294087b4aaf8b0c4d9dd3d2d474"
format_stamp: "Formatted at 2026-09-29 20:22:34 on dist-test-slave-zbnc"
server_key: "ab8f02f1f101f6f79bdf559e5126377e"
server_key_iv: "6f2e9fbaacfdb1a62c017d05fa63ddbd"
server_key_version: "encryptionkey@0"
I20260929 20:22:34.917690 19592 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790713339161492-17621-0/minicluster-data/ts-1/wal/instance:
uuid: "fc610294087b4aaf8b0c4d9dd3d2d474"
format_stamp: "Formatted at 2026-09-29 20:22:34 on dist-test-slave-zbnc"
server_key: "ab8f02f1f101f6f79bdf559e5126377e"
server_key_iv: "6f2e9fbaacfdb1a62c017d05fa63ddbd"
server_key_version: "encryptionkey@0"
I20260929 20:22:34.923272 19592 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260929 20:22:34.926723 19607 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:22:34.928146 19592 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 20:22:34.928333 19592 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790713339161492-17621-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790713339161492-17621-0/minicluster-data/ts-1/wal
uuid: "fc610294087b4aaf8b0c4d9dd3d2d474"
format_stamp: "Formatted at 2026-09-29 20:22:34 on dist-test-slave-zbnc"
server_key: "ab8f02f1f101f6f79bdf559e5126377e"
server_key_iv: "6f2e9fbaacfdb1a62c017d05fa63ddbd"
server_key_version: "encryptionkey@0"
I20260929 20:22:34.928551 19592 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790713339161492-17621-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790713339161492-17621-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790713339161492-17621-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:22:34.966609 19592 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:22:34.971340 19592 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:22:34.971654 19592 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:22:34.974372 19592 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:22:34.974515 19592 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:22:34.974651 19592 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:22:34.974740 19592 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 20:22:34 dist-test-slave-zbnc krb5kdc[19334](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713354, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.53.66@KRBTEST.COM for kudu/127.17.53.126@KRBTEST.COM
I20260929 20:22:34.998803 19617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:34.976498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.53.66:52481 (local address 127.17.53.126:42509)
0929 20:22:34.976944 (+ 446us) server_negotiation.cc:206] Beginning negotiation
0929 20:22:34.976951 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:22:34.977375 (+ 424us) server_negotiation.cc:407] Connection header received
0929 20:22:34.979871 (+ 2496us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:22:34.979879 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:22:34.979989 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:22:34.980115 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:22:34.983122 (+ 3007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:34.984184 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:34.986329 (+ 2145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:34.986776 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:34.986949 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:22:34.991870 (+ 4921us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:22:34.991877 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:22:34.991881 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:22:34.991929 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:22:34.995385 (+ 3456us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:22:34.996395 (+ 1010us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:22:34.996402 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:22:34.996407 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:22:34.996551 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:22:34.997011 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:22:34.997016 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:22:34.997019 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:22:34.997299 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:22:34.997455 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0929 20:22:34.998190 (+ 735us) server_negotiation.cc:299] Negotiation successful
0929 20:22:34.998414 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":330,"thread_start_us":1095,"threads_started":1}
I20260929 20:22:34.999944 19616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:34.975970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:42509 (local address 127.17.53.66:52481)
0929 20:22:34.977225 (+ 1255us) negotiation.cc:107] Waiting for socket to connect
0929 20:22:34.977266 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 20:22:34.979393 (+ 2127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:22:34.980451 (+ 1058us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:22:34.980463 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:22:34.981616 (+ 1153us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:22:34.982884 (+ 1268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:22:34.982926 (+ 42us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:22:34.984403 (+ 1477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:22:34.984406 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:22:34.986174 (+ 1768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:22:34.986187 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:22:34.987056 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:22:34.987064 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:22:34.987349 (+ 285us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:22:34.988133 (+ 784us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:22:34.988177 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:22:34.991519 (+ 3342us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:22:34.995537 (+ 4018us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:22:34.995546 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:22:34.995567 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:22:34.996222 (+ 655us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:22:34.996665 (+ 443us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:22:34.996670 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:22:34.996674 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:22:34.996887 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:22:34.997437 (+ 550us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:22:34.997445 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:22:34.997838 (+ 393us) client_negotiation.cc:769] Sending connection context
0929 20:22:34.998102 (+ 264us) client_negotiation.cc:240] Negotiation successful
0929 20:22:34.998648 (+ 546us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":568,"thread_start_us":533,"threads_started":1}
I20260929 20:22:35.022878 19617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:35.006616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.53.66:41355 (local address 127.17.53.126:42509)
0929 20:22:35.006911 (+ 295us) server_negotiation.cc:206] Beginning negotiation
0929 20:22:35.006920 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 20:22:35.007507 (+ 587us) server_negotiation.cc:407] Connection header received
0929 20:22:35.007785 (+ 278us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:22:35.007789 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:22:35.007878 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:22:35.007975 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:22:35.009497 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:35.010202 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:35.012757 (+ 2555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:35.013168 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:35.013280 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:22:35.016405 (+ 3125us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:22:35.016414 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:22:35.016421 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:22:35.016467 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:22:35.020124 (+ 3657us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:22:35.020838 (+ 714us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:22:35.020844 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:22:35.020847 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:22:35.020974 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:22:35.021359 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:22:35.021364 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:22:35.021367 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:22:35.021705 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:22:35.021870 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0929 20:22:35.022268 (+ 398us) server_negotiation.cc:299] Negotiation successful
0929 20:22:35.022498 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260929 20:22:35.022986 19671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:35.006502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:42509 (local address 127.17.53.66:41355)
0929 20:22:35.007366 (+ 864us) negotiation.cc:107] Waiting for socket to connect
0929 20:22:35.007392 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0929 20:22:35.007589 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:22:35.008042 (+ 453us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:22:35.008050 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:22:35.008823 (+ 773us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:22:35.009329 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:22:35.009347 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:22:35.010366 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:22:35.010374 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:22:35.012567 (+ 2193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:22:35.012589 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:22:35.013404 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:22:35.013414 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:22:35.013707 (+ 293us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:22:35.014497 (+ 790us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:22:35.014522 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:22:35.016079 (+ 1557us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:22:35.020275 (+ 4196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:22:35.020282 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:22:35.020289 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:22:35.020708 (+ 419us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:22:35.021104 (+ 396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:22:35.021108 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:22:35.021112 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:22:35.021238 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:22:35.021831 (+ 593us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:22:35.021836 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:22:35.022017 (+ 181us) client_negotiation.cc:769] Sending connection context
0929 20:22:35.022168 (+ 151us) client_negotiation.cc:240] Negotiation successful
0929 20:22:35.022523 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":651,"thread_start_us":480,"threads_started":1}
I20260929 20:22:35.036806 19592 rpc_server.cc:307] RPC server started. Bound to: 127.17.53.66:43673
I20260929 20:22:35.036854 19727 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.53.66:43673 every 8 connection(s)
I20260929 20:22:35.038103 19592 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790713339161492-17621-0/minicluster-data/ts-1/data/info.pb
I20260929 20:22:35.040184 19728 heartbeater.cc:347] Connected to a master server at 127.17.53.126:42509
I20260929 20:22:35.040426 19728 heartbeater.cc:464] Registering TS with master...
I20260929 20:22:35.041115 19728 heartbeater.cc:511] Master 127.17.53.126:42509 requested a full tablet report, sending...
I20260929 20:22:35.041867 17621 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu as pid 19592
I20260929 20:22:35.042075 17621 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790713339161492-17621-0/minicluster-data/ts-1/wal/instance
I20260929 20:22:35.042519 17621 external_mini_cluster.cc:1428] Setting key 81a528dbdb2bdcddb1f57fb47b0c1d54
I20260929 20:22:35.042632 19380 ts_manager.cc:194] Registered new tserver with Master: fc610294087b4aaf8b0c4d9dd3d2d474 (127.17.53.66:43673)
I20260929 20:22:35.043674 19380 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.53.66@KRBTEST.COM'} at 127.17.53.66:52481
WARNING: no policy specified for kudu/127.17.53.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.53.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.53.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.67.keytab.
Entry for principal kudu/127.17.53.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.67.keytab.
I20260929 20:22:35.086380 17621 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
/tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790713339161492-17621-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790713339161492-17621-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790713339161492-17621-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790713339161492-17621-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.53.67:0
--local_ip_for_outbound_sockets=127.17.53.67
--webserver_interface=127.17.53.67
--webserver_port=0
--tserver_master_addrs=127.17.53.126:42509
--keytab_file=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.67.keytab
--principal=kudu/127.17.53.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.53.84:39917
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790713339161492-17621-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790713339161492-17621-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790713339161492-17621-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:22:35.421000 19732 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:22:35.421295 19732 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:22:35.421466 19732 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:22:35.428835 19732 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:22:35.428957 19732 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:22:35.429121 19732 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260929 20:22:35.429198 19732 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 20:22:35.429329 19732 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.53.67
I20260929 20:22:35.441588 19732 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.53.84:39917
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790713339161492-17621-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790713339161492-17621-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.67.keytab
--principal=kudu/127.17.53.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.53.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790713339161492-17621-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.17.53.67
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.17.53.126:42509
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.53.67
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790713339161492-17621-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:22:35.443115 19732 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:22:35.444558 19732 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:22:35.458906 19740 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:22:35.459110 19738 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:22:35.459165 19737 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:22:35.460219 19732 server_base.cc:1034] running on GCE node
I20260929 20:22:35.460872 19732 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:22:35.461841 19732 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:22:35.463065 19732 hybrid_clock.cc:648] HybridClock initialized: now 1790713355463016 us; error 43 us; skew 500 ppm
I20260929 20:22:35.463383 19732 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:22:35 dist-test-slave-zbnc krb5kdc[19334](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713355, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.53.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:22:35.466447 19732 init.cc:377] Logged in from keytab as kudu/127.17.53.67@KRBTEST.COM (short username kudu)
I20260929 20:22:35.468021 19732 webserver.cc:460] Webserver started at http://127.17.53.67:44695/ using document root <none> and password file <none>
I20260929 20:22:35.468705 19732 fs_manager.cc:359] Metadata directory not provided
I20260929 20:22:35.468855 19732 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:22:35.469177 19732 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:22:35.472131 19732 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790713339161492-17621-0/minicluster-data/ts-2/data/instance:
uuid: "82f1641f4cb34d93a53010da960b9cc9"
format_stamp: "Formatted at 2026-09-29 20:22:35 on dist-test-slave-zbnc"
server_key: "9adf1e848540f3a4899201e66c22b7ef"
server_key_iv: "acc7e205f64d3ddc710edaa5081b6929"
server_key_version: "encryptionkey@0"
I20260929 20:22:35.472884 19732 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790713339161492-17621-0/minicluster-data/ts-2/wal/instance:
uuid: "82f1641f4cb34d93a53010da960b9cc9"
format_stamp: "Formatted at 2026-09-29 20:22:35 on dist-test-slave-zbnc"
server_key: "9adf1e848540f3a4899201e66c22b7ef"
server_key_iv: "acc7e205f64d3ddc710edaa5081b6929"
server_key_version: "encryptionkey@0"
I20260929 20:22:35.478174 19732 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260929 20:22:35.481689 19747 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:22:35.483093 19732 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 20:22:35.483259 19732 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790713339161492-17621-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790713339161492-17621-0/minicluster-data/ts-2/wal
uuid: "82f1641f4cb34d93a53010da960b9cc9"
format_stamp: "Formatted at 2026-09-29 20:22:35 on dist-test-slave-zbnc"
server_key: "9adf1e848540f3a4899201e66c22b7ef"
server_key_iv: "acc7e205f64d3ddc710edaa5081b6929"
server_key_version: "encryptionkey@0"
I20260929 20:22:35.483726 19732 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790713339161492-17621-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790713339161492-17621-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790713339161492-17621-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:22:35.485672 19588 heartbeater.cc:503] Master 127.17.53.126:42509 was elected leader, sending a full tablet report...
I20260929 20:22:35.507302 19732 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:22:35.511796 19732 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:22:35.512130 19732 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:22:35.515210 19732 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:22:35.515372 19732 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:22:35.515507 19732 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:22:35.515641 19732 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 20:22:35 dist-test-slave-zbnc krb5kdc[19334](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713355, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.53.67@KRBTEST.COM for kudu/127.17.53.126@KRBTEST.COM
I20260929 20:22:35.545562 19617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:35.515438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.53.67:45819 (local address 127.17.53.126:42509)
0929 20:22:35.515778 (+ 340us) server_negotiation.cc:206] Beginning negotiation
0929 20:22:35.515785 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:22:35.517234 (+ 1449us) server_negotiation.cc:407] Connection header received
0929 20:22:35.520400 (+ 3166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:22:35.520405 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:22:35.520479 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:22:35.520568 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:22:35.523567 (+ 2999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:35.525183 (+ 1616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:35.528991 (+ 3808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:35.529610 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:35.529785 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:22:35.536074 (+ 6289us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:22:35.536083 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:22:35.536090 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:22:35.536141 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:22:35.540432 (+ 4291us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:22:35.541900 (+ 1468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:22:35.541910 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:22:35.541917 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:22:35.542129 (+ 212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:22:35.543008 (+ 879us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:22:35.543015 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:22:35.543021 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:22:35.543343 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:22:35.543467 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0929 20:22:35.544801 (+ 1334us) server_negotiation.cc:299] Negotiation successful
0929 20:22:35.545089 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":196}
I20260929 20:22:35.547926 19756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:35.516110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:42509 (local address 127.17.53.67:45819)
0929 20:22:35.516999 (+ 889us) negotiation.cc:107] Waiting for socket to connect
0929 20:22:35.517083 (+ 84us) client_negotiation.cc:174] Beginning negotiation
0929 20:22:35.519969 (+ 2886us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:22:35.520896 (+ 927us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:22:35.520920 (+ 24us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:22:35.522027 (+ 1107us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:22:35.523247 (+ 1220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:22:35.523286 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:22:35.525545 (+ 2259us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:22:35.525556 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:22:35.528703 (+ 3147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:22:35.528724 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:22:35.529935 (+ 1211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:22:35.529947 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:22:35.530433 (+ 486us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:22:35.531396 (+ 963us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:22:35.531425 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:22:35.535701 (+ 4276us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:22:35.540664 (+ 4963us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:22:35.540680 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:22:35.540705 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:22:35.541643 (+ 938us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:22:35.542320 (+ 677us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:22:35.542329 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:22:35.542341 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:22:35.542750 (+ 409us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:22:35.543473 (+ 723us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:22:35.543485 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:22:35.544187 (+ 702us) client_negotiation.cc:769] Sending connection context
0929 20:22:35.544663 (+ 476us) client_negotiation.cc:240] Negotiation successful
0929 20:22:35.545546 (+ 883us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":584,"thread_start_us":334,"threads_started":1}
I20260929 20:22:35.566013 19732 rpc_server.cc:307] RPC server started. Bound to: 127.17.53.67:39893
I20260929 20:22:35.566046 19868 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.53.67:39893 every 8 connection(s)
I20260929 20:22:35.567777 19732 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790713339161492-17621-0/minicluster-data/ts-2/data/info.pb
I20260929 20:22:35.569808 19842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:35.554248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:42509 (local address 127.17.53.67:52939)
0929 20:22:35.555481 (+ 1233us) negotiation.cc:107] Waiting for socket to connect
0929 20:22:35.555508 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0929 20:22:35.555720 (+ 212us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:22:35.556108 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:22:35.556113 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:22:35.556613 (+ 500us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:22:35.556949 (+ 336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:22:35.556960 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:22:35.558317 (+ 1357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:22:35.558323 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:22:35.559678 (+ 1355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:22:35.559691 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:22:35.560900 (+ 1209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:22:35.560905 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:22:35.561039 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:22:35.561914 (+ 875us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:22:35.561933 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:22:35.563113 (+ 1180us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:22:35.566527 (+ 3414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:22:35.566533 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:22:35.566538 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:22:35.567133 (+ 595us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:22:35.567694 (+ 561us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:22:35.567702 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:22:35.567706 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:22:35.567870 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:22:35.568570 (+ 700us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:22:35.568575 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:22:35.568795 (+ 220us) client_negotiation.cc:769] Sending connection context
0929 20:22:35.568945 (+ 150us) client_negotiation.cc:240] Negotiation successful
0929 20:22:35.569330 (+ 385us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":996,"thread_start_us":531,"threads_started":1}
I20260929 20:22:35.569945 19617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:35.554106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.53.67:52939 (local address 127.17.53.126:42509)
0929 20:22:35.554468 (+ 362us) server_negotiation.cc:206] Beginning negotiation
0929 20:22:35.554477 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 20:22:35.555593 (+ 1116us) server_negotiation.cc:407] Connection header received
0929 20:22:35.555895 (+ 302us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:22:35.555900 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:22:35.555973 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:22:35.556076 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:22:35.557093 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:35.558178 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:35.559823 (+ 1645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:35.560475 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:35.560626 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:22:35.563358 (+ 2732us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:22:35.563363 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:22:35.563366 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:22:35.563395 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:22:35.566400 (+ 3005us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:22:35.567360 (+ 960us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:22:35.567364 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:22:35.567366 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:22:35.567514 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:22:35.568012 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:22:35.568017 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:22:35.568021 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:22:35.568432 (+ 411us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:22:35.568612 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0929 20:22:35.569130 (+ 518us) server_negotiation.cc:299] Negotiation successful
0929 20:22:35.569440 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":218}
I20260929 20:22:35.570760 19869 heartbeater.cc:347] Connected to a master server at 127.17.53.126:42509
I20260929 20:22:35.571058 19869 heartbeater.cc:464] Registering TS with master...
I20260929 20:22:35.571894 19869 heartbeater.cc:511] Master 127.17.53.126:42509 requested a full tablet report, sending...
I20260929 20:22:35.573554 19379 ts_manager.cc:194] Registered new tserver with Master: 82f1641f4cb34d93a53010da960b9cc9 (127.17.53.67:39893)
I20260929 20:22:35.574463 19379 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.53.67@KRBTEST.COM'} at 127.17.53.67:45819
I20260929 20:22:35.575260 17621 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu as pid 19732
I20260929 20:22:35.575431 17621 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790713339161492-17621-0/minicluster-data/ts-2/wal/instance
I20260929 20:22:35.576032 17621 external_mini_cluster.cc:1428] Setting key b0f534aeaf6ad98ea3b82bcc46089dc5
I20260929 20:22:35.578791 17621 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 29 20:22:35 dist-test-slave-zbnc krb5kdc[19334](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713355, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 29 20:22:35 dist-test-slave-zbnc krb5kdc[19334](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713355, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.53.126@KRBTEST.COM
I20260929 20:22:35.625659 19617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:35.612092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60614 (local address 127.17.53.126:42509)
0929 20:22:35.612344 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0929 20:22:35.612350 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:22:35.612589 (+ 239us) server_negotiation.cc:407] Connection header received
0929 20:22:35.612773 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:22:35.612777 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:22:35.612841 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:22:35.612931 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:22:35.614372 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:35.615212 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:35.616415 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:35.616809 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:35.616897 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:22:35.620930 (+ 4033us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:22:35.620935 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:22:35.620939 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:22:35.620972 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:22:35.623438 (+ 2466us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:22:35.624021 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:22:35.624025 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:22:35.624027 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:22:35.624142 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:22:35.624450 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:22:35.624453 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:22:35.624455 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:22:35.624697 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:22:35.624896 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0929 20:22:35.625139 (+ 243us) server_negotiation.cc:299] Negotiation successful
0929 20:22:35.625291 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260929 20:22:35.629858 19379 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60614:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 20:22:35.637415 19379 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 20:22:35.671684 19888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:35.651982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50368 (local address 127.17.53.67:39893)
0929 20:22:35.660990 (+ 9008us) server_negotiation.cc:206] Beginning negotiation
0929 20:22:35.661004 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 20:22:35.662758 (+ 1754us) server_negotiation.cc:407] Connection header received
0929 20:22:35.662906 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:22:35.662914 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:22:35.663222 (+ 308us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:22:35.663421 (+ 199us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:22:35.664704 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:35.666048 (+ 1344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:35.668798 (+ 2750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:35.670447 (+ 1649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:35.670593 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:22:35.670888 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0929 20:22:35.671134 (+ 246us) server_negotiation.cc:299] Negotiation successful
0929 20:22:35.671322 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8756,"thread_start_us":8497,"threads_started":1}
I20260929 20:22:35.671790 19887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:35.651407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.67:39893 (local address 127.0.0.1:50368)
0929 20:22:35.662573 (+ 11166us) negotiation.cc:107] Waiting for socket to connect
0929 20:22:35.662591 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 20:22:35.662728 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:22:35.663770 (+ 1042us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:22:35.663776 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:22:35.663805 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:22:35.664478 (+ 673us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:22:35.664488 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:22:35.666237 (+ 1749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:22:35.666242 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:22:35.668625 (+ 2383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:22:35.668638 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:22:35.670637 (+ 1999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:22:35.670642 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:22:35.670850 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:22:35.670955 (+ 105us) client_negotiation.cc:769] Sending connection context
0929 20:22:35.671099 (+ 144us) client_negotiation.cc:240] Negotiation successful
0929 20:22:35.671331 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11046,"mutex_wait_us":253,"thread_start_us":10902,"threads_started":1}
I20260929 20:22:35.672320 19886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:35.650182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36066 (local address 127.17.53.66:43673)
0929 20:22:35.661388 (+ 11206us) server_negotiation.cc:206] Beginning negotiation
0929 20:22:35.661401 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 20:22:35.662465 (+ 1064us) server_negotiation.cc:407] Connection header received
0929 20:22:35.663158 (+ 693us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:22:35.663167 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:22:35.663402 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:22:35.663555 (+ 153us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:22:35.664690 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:35.666049 (+ 1359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:35.669492 (+ 3443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:35.670933 (+ 1441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:35.671056 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:22:35.671278 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0929 20:22:35.671858 (+ 580us) server_negotiation.cc:299] Negotiation successful
0929 20:22:35.672027 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10958,"thread_start_us":11214,"threads_started":1}
I20260929 20:22:35.672330 19882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:35.649181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.65:43187 (local address 127.0.0.1:57486)
0929 20:22:35.650904 (+ 1723us) negotiation.cc:107] Waiting for socket to connect
0929 20:22:35.650981 (+ 77us) client_negotiation.cc:174] Beginning negotiation
0929 20:22:35.651193 (+ 212us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:22:35.663602 (+ 12409us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:22:35.663610 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:22:35.663642 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:22:35.664049 (+ 407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:22:35.664063 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:22:35.666173 (+ 2110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:22:35.666178 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:22:35.669307 (+ 3129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:22:35.669322 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:22:35.671170 (+ 1848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:22:35.671177 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:22:35.671381 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:22:35.671425 (+ 44us) client_negotiation.cc:769] Sending connection context
0929 20:22:35.671494 (+ 69us) client_negotiation.cc:240] Negotiation successful
0929 20:22:35.671635 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1376,"mutex_wait_us":272,"thread_start_us":1181,"threads_started":1}
I20260929 20:22:35.673189 19883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:35.649428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57486 (local address 127.17.53.65:43187)
0929 20:22:35.662924 (+ 13496us) server_negotiation.cc:206] Beginning negotiation
0929 20:22:35.662932 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 20:22:35.662958 (+ 26us) server_negotiation.cc:407] Connection header received
0929 20:22:35.663081 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:22:35.663089 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:22:35.663351 (+ 262us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:22:35.664171 (+ 820us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:22:35.664397 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:35.665994 (+ 1597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:35.669497 (+ 3503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:35.670933 (+ 1436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:35.671885 (+ 952us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:22:35.672647 (+ 762us) server_negotiation.cc:1035] Waiting for connection context
0929 20:22:35.672723 (+ 76us) server_negotiation.cc:299] Negotiation successful
0929 20:22:35.672903 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13327,"thread_start_us":14019,"threads_started":1}
I20260929 20:22:35.673292 19884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:35.649407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.66:43673 (local address 127.0.0.1:36066)
0929 20:22:35.662277 (+ 12870us) negotiation.cc:107] Waiting for socket to connect
0929 20:22:35.662306 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0929 20:22:35.662966 (+ 660us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:22:35.663792 (+ 826us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:22:35.663795 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:22:35.663819 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:22:35.664445 (+ 626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:22:35.664465 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:22:35.666223 (+ 1758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:22:35.666227 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:22:35.669307 (+ 3080us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:22:35.669322 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:22:35.671475 (+ 2153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:22:35.671484 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:22:35.671658 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:22:35.671684 (+ 26us) client_negotiation.cc:769] Sending connection context
0929 20:22:35.672790 (+ 1106us) client_negotiation.cc:240] Negotiation successful
0929 20:22:35.672916 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12650,"mutex_wait_us":177,"thread_start_us":12424,"threads_started":1}
I20260929 20:22:35.675927 19798 tablet_service.cc:1467] Processing CreateTablet for tablet 36302175c35d448ebbad5d77b50dd4ab (DEFAULT_TABLE table=test-table [id=800b83317dd04ad4ad9dab9580e5c14d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:22:35.676772 19658 tablet_service.cc:1467] Processing CreateTablet for tablet 36302175c35d448ebbad5d77b50dd4ab (DEFAULT_TABLE table=test-table [id=800b83317dd04ad4ad9dab9580e5c14d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:22:35.676767 19518 tablet_service.cc:1467] Processing CreateTablet for tablet 36302175c35d448ebbad5d77b50dd4ab (DEFAULT_TABLE table=test-table [id=800b83317dd04ad4ad9dab9580e5c14d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:22:35.677457 19798 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 36302175c35d448ebbad5d77b50dd4ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:22:35.678049 19518 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 36302175c35d448ebbad5d77b50dd4ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:22:35.678062 19658 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 36302175c35d448ebbad5d77b50dd4ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:22:35.689971 19889 tablet_bootstrap.cc:492] T 36302175c35d448ebbad5d77b50dd4ab P 82f1641f4cb34d93a53010da960b9cc9: Bootstrap starting.
I20260929 20:22:35.692406 19891 tablet_bootstrap.cc:492] T 36302175c35d448ebbad5d77b50dd4ab P fc610294087b4aaf8b0c4d9dd3d2d474: Bootstrap starting.
I20260929 20:22:35.693186 19890 tablet_bootstrap.cc:492] T 36302175c35d448ebbad5d77b50dd4ab P ba9328e853e140d6b28c75c85f061c69: Bootstrap starting.
I20260929 20:22:35.693950 19889 tablet_bootstrap.cc:654] T 36302175c35d448ebbad5d77b50dd4ab P 82f1641f4cb34d93a53010da960b9cc9: Neither blocks nor log segments found. Creating new log.
I20260929 20:22:35.695334 19889 log.cc:824] T 36302175c35d448ebbad5d77b50dd4ab P 82f1641f4cb34d93a53010da960b9cc9: Log is configured to *not* fsync() on all Append() calls
I20260929 20:22:35.697042 19890 tablet_bootstrap.cc:654] T 36302175c35d448ebbad5d77b50dd4ab P ba9328e853e140d6b28c75c85f061c69: Neither blocks nor log segments found. Creating new log.
I20260929 20:22:35.697944 19891 tablet_bootstrap.cc:654] T 36302175c35d448ebbad5d77b50dd4ab P fc610294087b4aaf8b0c4d9dd3d2d474: Neither blocks nor log segments found. Creating new log.
I20260929 20:22:35.698657 19890 log.cc:824] T 36302175c35d448ebbad5d77b50dd4ab P ba9328e853e140d6b28c75c85f061c69: Log is configured to *not* fsync() on all Append() calls
I20260929 20:22:35.698980 19891 log.cc:824] T 36302175c35d448ebbad5d77b50dd4ab P fc610294087b4aaf8b0c4d9dd3d2d474: Log is configured to *not* fsync() on all Append() calls
I20260929 20:22:35.701929 19891 tablet_bootstrap.cc:492] T 36302175c35d448ebbad5d77b50dd4ab P fc610294087b4aaf8b0c4d9dd3d2d474: No bootstrap required, opened a new log
I20260929 20:22:35.701963 19889 tablet_bootstrap.cc:492] T 36302175c35d448ebbad5d77b50dd4ab P 82f1641f4cb34d93a53010da960b9cc9: No bootstrap required, opened a new log
I20260929 20:22:35.701967 19890 tablet_bootstrap.cc:492] T 36302175c35d448ebbad5d77b50dd4ab P ba9328e853e140d6b28c75c85f061c69: No bootstrap required, opened a new log
I20260929 20:22:35.702215 19891 ts_tablet_manager.cc:1397] T 36302175c35d448ebbad5d77b50dd4ab P fc610294087b4aaf8b0c4d9dd3d2d474: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.000s
I20260929 20:22:35.702226 19889 ts_tablet_manager.cc:1397] T 36302175c35d448ebbad5d77b50dd4ab P 82f1641f4cb34d93a53010da960b9cc9: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.003s
I20260929 20:22:35.702226 19890 ts_tablet_manager.cc:1397] T 36302175c35d448ebbad5d77b50dd4ab P ba9328e853e140d6b28c75c85f061c69: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.003s
I20260929 20:22:35.710716 19891 raft_consensus.cc:348] T 36302175c35d448ebbad5d77b50dd4ab P fc610294087b4aaf8b0c4d9dd3d2d474 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba9328e853e140d6b28c75c85f061c69" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 43187 } } peers { permanent_uuid: "fc610294087b4aaf8b0c4d9dd3d2d474" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 43673 } } peers { permanent_uuid: "82f1641f4cb34d93a53010da960b9cc9" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 39893 } }
I20260929 20:22:35.711160 19891 raft_consensus.cc:374] T 36302175c35d448ebbad5d77b50dd4ab P fc610294087b4aaf8b0c4d9dd3d2d474 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:22:35.711334 19891 raft_consensus.cc:729] T 36302175c35d448ebbad5d77b50dd4ab P fc610294087b4aaf8b0c4d9dd3d2d474 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc610294087b4aaf8b0c4d9dd3d2d474, State: Initialized, Role: FOLLOWER
I20260929 20:22:35.711970 19891 consensus_queue.cc:260] T 36302175c35d448ebbad5d77b50dd4ab P fc610294087b4aaf8b0c4d9dd3d2d474 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba9328e853e140d6b28c75c85f061c69" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 43187 } } peers { permanent_uuid: "fc610294087b4aaf8b0c4d9dd3d2d474" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 43673 } } peers { permanent_uuid: "82f1641f4cb34d93a53010da960b9cc9" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 39893 } }
I20260929 20:22:35.713078 19728 heartbeater.cc:503] Master 127.17.53.126:42509 was elected leader, sending a full tablet report...
I20260929 20:22:35.713163 19890 raft_consensus.cc:348] T 36302175c35d448ebbad5d77b50dd4ab P ba9328e853e140d6b28c75c85f061c69 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba9328e853e140d6b28c75c85f061c69" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 43187 } } peers { permanent_uuid: "fc610294087b4aaf8b0c4d9dd3d2d474" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 43673 } } peers { permanent_uuid: "82f1641f4cb34d93a53010da960b9cc9" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 39893 } }
I20260929 20:22:35.713165 19889 raft_consensus.cc:348] T 36302175c35d448ebbad5d77b50dd4ab P 82f1641f4cb34d93a53010da960b9cc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba9328e853e140d6b28c75c85f061c69" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 43187 } } peers { permanent_uuid: "fc610294087b4aaf8b0c4d9dd3d2d474" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 43673 } } peers { permanent_uuid: "82f1641f4cb34d93a53010da960b9cc9" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 39893 } }
I20260929 20:22:35.713603 19889 raft_consensus.cc:374] T 36302175c35d448ebbad5d77b50dd4ab P 82f1641f4cb34d93a53010da960b9cc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:22:35.713600 19890 raft_consensus.cc:374] T 36302175c35d448ebbad5d77b50dd4ab P ba9328e853e140d6b28c75c85f061c69 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:22:35.713692 19890 raft_consensus.cc:729] T 36302175c35d448ebbad5d77b50dd4ab P ba9328e853e140d6b28c75c85f061c69 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba9328e853e140d6b28c75c85f061c69, State: Initialized, Role: FOLLOWER
I20260929 20:22:35.713692 19889 raft_consensus.cc:729] T 36302175c35d448ebbad5d77b50dd4ab P 82f1641f4cb34d93a53010da960b9cc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82f1641f4cb34d93a53010da960b9cc9, State: Initialized, Role: FOLLOWER
I20260929 20:22:35.713778 19891 ts_tablet_manager.cc:1428] T 36302175c35d448ebbad5d77b50dd4ab P fc610294087b4aaf8b0c4d9dd3d2d474: Time spent starting tablet: real 0.011s user 0.008s sys 0.005s
I20260929 20:22:35.714268 19890 consensus_queue.cc:260] T 36302175c35d448ebbad5d77b50dd4ab P ba9328e853e140d6b28c75c85f061c69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba9328e853e140d6b28c75c85f061c69" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 43187 } } peers { permanent_uuid: "fc610294087b4aaf8b0c4d9dd3d2d474" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 43673 } } peers { permanent_uuid: "82f1641f4cb34d93a53010da960b9cc9" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 39893 } }
I20260929 20:22:35.714255 19889 consensus_queue.cc:260] T 36302175c35d448ebbad5d77b50dd4ab P 82f1641f4cb34d93a53010da960b9cc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba9328e853e140d6b28c75c85f061c69" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 43187 } } peers { permanent_uuid: "fc610294087b4aaf8b0c4d9dd3d2d474" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 43673 } } peers { permanent_uuid: "82f1641f4cb34d93a53010da960b9cc9" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 39893 } }
I20260929 20:22:35.715979 19869 heartbeater.cc:503] Master 127.17.53.126:42509 was elected leader, sending a full tablet report...
I20260929 20:22:35.716534 19889 ts_tablet_manager.cc:1428] T 36302175c35d448ebbad5d77b50dd4ab P 82f1641f4cb34d93a53010da960b9cc9: Time spent starting tablet: real 0.014s user 0.012s sys 0.003s
I20260929 20:22:35.717770 19890 ts_tablet_manager.cc:1428] T 36302175c35d448ebbad5d77b50dd4ab P ba9328e853e140d6b28c75c85f061c69: Time spent starting tablet: real 0.015s user 0.014s sys 0.002s
W20260929 20:22:35.728977 19589 tablet.cc:2369] T 36302175c35d448ebbad5d77b50dd4ab P ba9328e853e140d6b28c75c85f061c69: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 20:22:35.790303 19729 tablet.cc:2369] T 36302175c35d448ebbad5d77b50dd4ab P fc610294087b4aaf8b0c4d9dd3d2d474: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 20:22:35.820961 19870 tablet.cc:2369] T 36302175c35d448ebbad5d77b50dd4ab P 82f1641f4cb34d93a53010da960b9cc9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 20:22:35.834347 19896 raft_consensus.cc:482] T 36302175c35d448ebbad5d77b50dd4ab P ba9328e853e140d6b28c75c85f061c69 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 20:22:35.834684 19896 raft_consensus.cc:504] T 36302175c35d448ebbad5d77b50dd4ab P ba9328e853e140d6b28c75c85f061c69 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba9328e853e140d6b28c75c85f061c69" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 43187 } } peers { permanent_uuid: "fc610294087b4aaf8b0c4d9dd3d2d474" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 43673 } } peers { permanent_uuid: "82f1641f4cb34d93a53010da960b9cc9" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 39893 } }
I20260929 20:22:35.836681 19896 leader_election.cc:288] T 36302175c35d448ebbad5d77b50dd4ab P ba9328e853e140d6b28c75c85f061c69 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fc610294087b4aaf8b0c4d9dd3d2d474 (127.17.53.66:43673), 82f1641f4cb34d93a53010da960b9cc9 (127.17.53.67:39893)
I20260929 20:22:35.843767 19886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:35.837112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.53.65:52295 (local address 127.17.53.66:43673)
0929 20:22:35.837327 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0929 20:22:35.837333 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:22:35.838241 (+ 908us) server_negotiation.cc:407] Connection header received
0929 20:22:35.838470 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:22:35.838475 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:22:35.838542 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:22:35.838649 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:22:35.839387 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:35.840106 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:35.842060 (+ 1954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:35.842818 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:35.843033 (+ 215us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:22:35.843119 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0929 20:22:35.843375 (+ 256us) server_negotiation.cc:299] Negotiation successful
0929 20:22:35.843500 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260929 20:22:35.843961 19898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:35.836861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.66:43673 (local address 127.17.53.65:52295)
0929 20:22:35.838125 (+ 1264us) negotiation.cc:107] Waiting for socket to connect
0929 20:22:35.838144 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 20:22:35.838314 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:22:35.838706 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:22:35.838711 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:22:35.838766 (+ 55us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:22:35.839252 (+ 486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:22:35.839262 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:22:35.840245 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:22:35.840251 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:22:35.841888 (+ 1637us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:22:35.841900 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:22:35.843046 (+ 1146us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:22:35.843052 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:22:35.843223 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:22:35.843247 (+ 24us) client_negotiation.cc:769] Sending connection context
0929 20:22:35.843352 (+ 105us) client_negotiation.cc:240] Negotiation successful
0929 20:22:35.843491 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1091,"mutex_wait_us":41,"spinlock_wait_cycles":5632,"thread_start_us":741,"threads_started":1}
I20260929 20:22:35.844264 19888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:35.837401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.53.65:59225 (local address 127.17.53.67:39893)
0929 20:22:35.837584 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0929 20:22:35.837592 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 20:22:35.838256 (+ 664us) server_negotiation.cc:407] Connection header received
0929 20:22:35.838510 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:22:35.838522 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:22:35.838582 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:22:35.838675 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:22:35.839378 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:35.840086 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:35.842291 (+ 2205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:35.843249 (+ 958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:35.843460 (+ 211us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:22:35.843541 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0929 20:22:35.843845 (+ 304us) server_negotiation.cc:299] Negotiation successful
0929 20:22:35.843982 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260929 20:22:35.844455 19899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:35.837171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.67:39893 (local address 127.17.53.65:59225)
0929 20:22:35.838126 (+ 955us) negotiation.cc:107] Waiting for socket to connect
0929 20:22:35.838141 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 20:22:35.838366 (+ 225us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:22:35.838692 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:22:35.838697 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:22:35.838763 (+ 66us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:22:35.839206 (+ 443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:22:35.839219 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:22:35.840214 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:22:35.840222 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:22:35.842097 (+ 1875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:22:35.842111 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:22:35.843489 (+ 1378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:22:35.843496 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:22:35.843674 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:22:35.843700 (+ 26us) client_negotiation.cc:769] Sending connection context
0929 20:22:35.843828 (+ 128us) client_negotiation.cc:240] Negotiation successful
0929 20:22:35.843974 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":842,"mutex_wait_us":121,"spinlock_wait_cycles":97280,"thread_start_us":546,"threads_started":1}
I20260929 20:22:35.845387 19683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36302175c35d448ebbad5d77b50dd4ab" candidate_uuid: "ba9328e853e140d6b28c75c85f061c69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc610294087b4aaf8b0c4d9dd3d2d474" is_pre_election: true
I20260929 20:22:35.845618 19819 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36302175c35d448ebbad5d77b50dd4ab" candidate_uuid: "ba9328e853e140d6b28c75c85f061c69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82f1641f4cb34d93a53010da960b9cc9" is_pre_election: true
I20260929 20:22:35.846063 19683 raft_consensus.cc:2454] T 36302175c35d448ebbad5d77b50dd4ab P fc610294087b4aaf8b0c4d9dd3d2d474 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ba9328e853e140d6b28c75c85f061c69 in term 0.
I20260929 20:22:35.846189 19819 raft_consensus.cc:2454] T 36302175c35d448ebbad5d77b50dd4ab P 82f1641f4cb34d93a53010da960b9cc9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ba9328e853e140d6b28c75c85f061c69 in term 0.
I20260929 20:22:35.846927 19467 leader_election.cc:302] T 36302175c35d448ebbad5d77b50dd4ab P ba9328e853e140d6b28c75c85f061c69 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 82f1641f4cb34d93a53010da960b9cc9, ba9328e853e140d6b28c75c85f061c69, fc610294087b4aaf8b0c4d9dd3d2d474; no voters:
I20260929 20:22:35.847419 19896 raft_consensus.cc:2781] T 36302175c35d448ebbad5d77b50dd4ab P ba9328e853e140d6b28c75c85f061c69 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 20:22:35.847580 19896 raft_consensus.cc:482] T 36302175c35d448ebbad5d77b50dd4ab P ba9328e853e140d6b28c75c85f061c69 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 20:22:35.847705 19896 raft_consensus.cc:3037] T 36302175c35d448ebbad5d77b50dd4ab P ba9328e853e140d6b28c75c85f061c69 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:22:35.849963 19896 raft_consensus.cc:504] T 36302175c35d448ebbad5d77b50dd4ab P ba9328e853e140d6b28c75c85f061c69 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba9328e853e140d6b28c75c85f061c69" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 43187 } } peers { permanent_uuid: "fc610294087b4aaf8b0c4d9dd3d2d474" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 43673 } } peers { permanent_uuid: "82f1641f4cb34d93a53010da960b9cc9" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 39893 } }
I20260929 20:22:35.850643 19896 leader_election.cc:288] T 36302175c35d448ebbad5d77b50dd4ab P ba9328e853e140d6b28c75c85f061c69 [CANDIDATE]: Term 1 election: Requested vote from peers fc610294087b4aaf8b0c4d9dd3d2d474 (127.17.53.66:43673), 82f1641f4cb34d93a53010da960b9cc9 (127.17.53.67:39893)
I20260929 20:22:35.851191 19819 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36302175c35d448ebbad5d77b50dd4ab" candidate_uuid: "ba9328e853e140d6b28c75c85f061c69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82f1641f4cb34d93a53010da960b9cc9"
I20260929 20:22:35.851231 19683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36302175c35d448ebbad5d77b50dd4ab" candidate_uuid: "ba9328e853e140d6b28c75c85f061c69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc610294087b4aaf8b0c4d9dd3d2d474"
I20260929 20:22:35.851405 19819 raft_consensus.cc:3037] T 36302175c35d448ebbad5d77b50dd4ab P 82f1641f4cb34d93a53010da960b9cc9 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:22:35.851459 19683 raft_consensus.cc:3037] T 36302175c35d448ebbad5d77b50dd4ab P fc610294087b4aaf8b0c4d9dd3d2d474 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:22:35.854698 19683 raft_consensus.cc:2454] T 36302175c35d448ebbad5d77b50dd4ab P fc610294087b4aaf8b0c4d9dd3d2d474 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ba9328e853e140d6b28c75c85f061c69 in term 1.
I20260929 20:22:35.854698 19819 raft_consensus.cc:2454] T 36302175c35d448ebbad5d77b50dd4ab P 82f1641f4cb34d93a53010da960b9cc9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ba9328e853e140d6b28c75c85f061c69 in term 1.
I20260929 20:22:35.855140 19469 leader_election.cc:302] T 36302175c35d448ebbad5d77b50dd4ab P ba9328e853e140d6b28c75c85f061c69 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 82f1641f4cb34d93a53010da960b9cc9, ba9328e853e140d6b28c75c85f061c69; no voters:
I20260929 20:22:35.855482 19896 raft_consensus.cc:2781] T 36302175c35d448ebbad5d77b50dd4ab P ba9328e853e140d6b28c75c85f061c69 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:22:35.855926 19896 raft_consensus.cc:686] T 36302175c35d448ebbad5d77b50dd4ab P ba9328e853e140d6b28c75c85f061c69 [term 1 LEADER]: Becoming Leader. State: Replica: ba9328e853e140d6b28c75c85f061c69, State: Running, Role: LEADER
I20260929 20:22:35.856444 19896 consensus_queue.cc:237] T 36302175c35d448ebbad5d77b50dd4ab P ba9328e853e140d6b28c75c85f061c69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba9328e853e140d6b28c75c85f061c69" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 43187 } } peers { permanent_uuid: "fc610294087b4aaf8b0c4d9dd3d2d474" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 43673 } } peers { permanent_uuid: "82f1641f4cb34d93a53010da960b9cc9" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 39893 } }
I20260929 20:22:35.862154 19379 catalog_manager.cc:5495] T 36302175c35d448ebbad5d77b50dd4ab P ba9328e853e140d6b28c75c85f061c69 reported cstate change: term changed from 0 to 1, leader changed from <none> to ba9328e853e140d6b28c75c85f061c69 (127.17.53.65). New cstate: current_term: 1 leader_uuid: "ba9328e853e140d6b28c75c85f061c69" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba9328e853e140d6b28c75c85f061c69" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 43187 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fc610294087b4aaf8b0c4d9dd3d2d474" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 43673 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "82f1641f4cb34d93a53010da960b9cc9" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 39893 } health_report { overall_health: UNKNOWN } } }
Sep 29 20:22:35 dist-test-slave-zbnc krb5kdc[19334](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713353, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.53.126@KRBTEST.COM for kudu/127.17.53.126@KRBTEST.COM
I20260929 20:22:35.953667 19907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:35.937330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:42509 (local address 127.0.0.1:60620)
0929 20:22:35.937908 (+ 578us) negotiation.cc:107] Waiting for socket to connect
0929 20:22:35.937925 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 20:22:35.938049 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:22:35.938364 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:22:35.938368 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:22:35.939153 (+ 785us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:22:35.939423 (+ 270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:22:35.939432 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:22:35.940387 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:22:35.940392 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:22:35.941469 (+ 1077us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:22:35.941480 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:22:35.942149 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:22:35.942153 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:22:35.942264 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:22:35.943019 (+ 755us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:22:35.943047 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:22:35.945940 (+ 2893us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:22:35.950141 (+ 4201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:22:35.950153 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:22:35.950182 (+ 29us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:22:35.950745 (+ 563us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:22:35.951729 (+ 984us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:22:35.951733 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:22:35.951735 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:22:35.951933 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:22:35.952585 (+ 652us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:22:35.952592 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:22:35.952799 (+ 207us) client_negotiation.cc:769] Sending connection context
0929 20:22:35.952946 (+ 147us) client_negotiation.cc:240] Negotiation successful
0929 20:22:35.953248 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":306,"threads_started":1}
I20260929 20:22:35.953739 19617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:35.937239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60620 (local address 127.17.53.126:42509)
0929 20:22:35.937494 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0929 20:22:35.937503 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 20:22:35.938013 (+ 510us) server_negotiation.cc:407] Connection header received
0929 20:22:35.938151 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:22:35.938155 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:22:35.938213 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:22:35.938310 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:22:35.939551 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:35.940236 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:35.941603 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:35.941990 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:35.942079 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:22:35.946153 (+ 4074us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:22:35.946159 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:22:35.946163 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:22:35.946205 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:22:35.949993 (+ 3788us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:22:35.950951 (+ 958us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:22:35.950959 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:22:35.950963 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:22:35.951143 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:22:35.952041 (+ 898us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:22:35.952046 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:22:35.952049 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:22:35.952441 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:22:35.952642 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0929 20:22:35.953071 (+ 429us) server_negotiation.cc:299] Negotiation successful
0929 20:22:35.953406 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":157,"server-negotiator.queue_time_us":82}
I20260929 20:22:35.958686 19379 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.17.53.126@KRBTEST.COM'} at 127.0.0.1:60620:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260929 20:22:35.960002 19379 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 20:22:35.966292 19518 tablet_service.cc:1467] Processing CreateTablet for tablet a524ff3c47bd4b0bbccfea6f3b0305e7 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=da84811e19c74f599231d0bc390cc557]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 20:22:35.967263 19518 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a524ff3c47bd4b0bbccfea6f3b0305e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:22:35.966634 19658 tablet_service.cc:1467] Processing CreateTablet for tablet a524ff3c47bd4b0bbccfea6f3b0305e7 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=da84811e19c74f599231d0bc390cc557]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 20:22:35.966634 19798 tablet_service.cc:1467] Processing CreateTablet for tablet a524ff3c47bd4b0bbccfea6f3b0305e7 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=da84811e19c74f599231d0bc390cc557]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 20:22:35.967609 19658 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a524ff3c47bd4b0bbccfea6f3b0305e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:22:35.967609 19798 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a524ff3c47bd4b0bbccfea6f3b0305e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:22:35.973140 19890 tablet_bootstrap.cc:492] T a524ff3c47bd4b0bbccfea6f3b0305e7 P ba9328e853e140d6b28c75c85f061c69: Bootstrap starting.
I20260929 20:22:35.974555 19890 tablet_bootstrap.cc:654] T a524ff3c47bd4b0bbccfea6f3b0305e7 P ba9328e853e140d6b28c75c85f061c69: Neither blocks nor log segments found. Creating new log.
I20260929 20:22:35.974924 19889 tablet_bootstrap.cc:492] T a524ff3c47bd4b0bbccfea6f3b0305e7 P 82f1641f4cb34d93a53010da960b9cc9: Bootstrap starting.
I20260929 20:22:35.975771 19891 tablet_bootstrap.cc:492] T a524ff3c47bd4b0bbccfea6f3b0305e7 P fc610294087b4aaf8b0c4d9dd3d2d474: Bootstrap starting.
I20260929 20:22:35.976537 19889 tablet_bootstrap.cc:654] T a524ff3c47bd4b0bbccfea6f3b0305e7 P 82f1641f4cb34d93a53010da960b9cc9: Neither blocks nor log segments found. Creating new log.
I20260929 20:22:35.977226 19890 tablet_bootstrap.cc:492] T a524ff3c47bd4b0bbccfea6f3b0305e7 P ba9328e853e140d6b28c75c85f061c69: No bootstrap required, opened a new log
I20260929 20:22:35.977231 19891 tablet_bootstrap.cc:654] T a524ff3c47bd4b0bbccfea6f3b0305e7 P fc610294087b4aaf8b0c4d9dd3d2d474: Neither blocks nor log segments found. Creating new log.
I20260929 20:22:35.977355 19890 ts_tablet_manager.cc:1397] T a524ff3c47bd4b0bbccfea6f3b0305e7 P ba9328e853e140d6b28c75c85f061c69: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20260929 20:22:35.978085 19890 raft_consensus.cc:348] T a524ff3c47bd4b0bbccfea6f3b0305e7 P ba9328e853e140d6b28c75c85f061c69 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba9328e853e140d6b28c75c85f061c69" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 43187 } } peers { permanent_uuid: "fc610294087b4aaf8b0c4d9dd3d2d474" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 43673 } } peers { permanent_uuid: "82f1641f4cb34d93a53010da960b9cc9" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 39893 } }
I20260929 20:22:35.978369 19890 raft_consensus.cc:374] T a524ff3c47bd4b0bbccfea6f3b0305e7 P ba9328e853e140d6b28c75c85f061c69 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:22:35.978475 19890 raft_consensus.cc:729] T a524ff3c47bd4b0bbccfea6f3b0305e7 P ba9328e853e140d6b28c75c85f061c69 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba9328e853e140d6b28c75c85f061c69, State: Initialized, Role: FOLLOWER
I20260929 20:22:35.978904 19889 tablet_bootstrap.cc:492] T a524ff3c47bd4b0bbccfea6f3b0305e7 P 82f1641f4cb34d93a53010da960b9cc9: No bootstrap required, opened a new log
I20260929 20:22:35.978708 19890 consensus_queue.cc:260] T a524ff3c47bd4b0bbccfea6f3b0305e7 P ba9328e853e140d6b28c75c85f061c69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba9328e853e140d6b28c75c85f061c69" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 43187 } } peers { permanent_uuid: "fc610294087b4aaf8b0c4d9dd3d2d474" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 43673 } } peers { permanent_uuid: "82f1641f4cb34d93a53010da960b9cc9" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 39893 } }
I20260929 20:22:35.979040 19889 ts_tablet_manager.cc:1397] T a524ff3c47bd4b0bbccfea6f3b0305e7 P 82f1641f4cb34d93a53010da960b9cc9: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20260929 20:22:35.979187 19896 tablet_replica.cc:439] T a524ff3c47bd4b0bbccfea6f3b0305e7 P ba9328e853e140d6b28c75c85f061c69: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba9328e853e140d6b28c75c85f061c69" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 43187 } } peers { permanent_uuid: "fc610294087b4aaf8b0c4d9dd3d2d474" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 43673 } } peers { permanent_uuid: "82f1641f4cb34d93a53010da960b9cc9" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 39893 } } }
I20260929 20:22:35.979460 19896 tablet_replica.cc:442] T a524ff3c47bd4b0bbccfea6f3b0305e7 P ba9328e853e140d6b28c75c85f061c69: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 20:22:35.979820 19889 raft_consensus.cc:348] T a524ff3c47bd4b0bbccfea6f3b0305e7 P 82f1641f4cb34d93a53010da960b9cc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba9328e853e140d6b28c75c85f061c69" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 43187 } } peers { permanent_uuid: "fc610294087b4aaf8b0c4d9dd3d2d474" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 43673 } } peers { permanent_uuid: "82f1641f4cb34d93a53010da960b9cc9" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 39893 } }
I20260929 20:22:35.980057 19889 raft_consensus.cc:374] T a524ff3c47bd4b0bbccfea6f3b0305e7 P 82f1641f4cb34d93a53010da960b9cc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:22:35.980163 19889 raft_consensus.cc:729] T a524ff3c47bd4b0bbccfea6f3b0305e7 P 82f1641f4cb34d93a53010da960b9cc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82f1641f4cb34d93a53010da960b9cc9, State: Initialized, Role: FOLLOWER
I20260929 20:22:35.980494 19890 ts_tablet_manager.cc:1428] T a524ff3c47bd4b0bbccfea6f3b0305e7 P ba9328e853e140d6b28c75c85f061c69: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
I20260929 20:22:35.980399 19889 consensus_queue.cc:260] T a524ff3c47bd4b0bbccfea6f3b0305e7 P 82f1641f4cb34d93a53010da960b9cc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba9328e853e140d6b28c75c85f061c69" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 43187 } } peers { permanent_uuid: "fc610294087b4aaf8b0c4d9dd3d2d474" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 43673 } } peers { permanent_uuid: "82f1641f4cb34d93a53010da960b9cc9" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 39893 } }
I20260929 20:22:35.980966 19889 ts_tablet_manager.cc:1428] T a524ff3c47bd4b0bbccfea6f3b0305e7 P 82f1641f4cb34d93a53010da960b9cc9: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260929 20:22:35.981184 19897 tablet_replica.cc:439] T a524ff3c47bd4b0bbccfea6f3b0305e7 P 82f1641f4cb34d93a53010da960b9cc9: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba9328e853e140d6b28c75c85f061c69" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 43187 } } peers { permanent_uuid: "fc610294087b4aaf8b0c4d9dd3d2d474" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 43673 } } peers { permanent_uuid: "82f1641f4cb34d93a53010da960b9cc9" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 39893 } } }
I20260929 20:22:35.981581 19897 tablet_replica.cc:442] T a524ff3c47bd4b0bbccfea6f3b0305e7 P 82f1641f4cb34d93a53010da960b9cc9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 20:22:35.983722 19891 tablet_bootstrap.cc:492] T a524ff3c47bd4b0bbccfea6f3b0305e7 P fc610294087b4aaf8b0c4d9dd3d2d474: No bootstrap required, opened a new log
I20260929 20:22:35.983871 19891 ts_tablet_manager.cc:1397] T a524ff3c47bd4b0bbccfea6f3b0305e7 P fc610294087b4aaf8b0c4d9dd3d2d474: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.004s
I20260929 20:22:35.984531 19891 raft_consensus.cc:348] T a524ff3c47bd4b0bbccfea6f3b0305e7 P fc610294087b4aaf8b0c4d9dd3d2d474 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba9328e853e140d6b28c75c85f061c69" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 43187 } } peers { permanent_uuid: "fc610294087b4aaf8b0c4d9dd3d2d474" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 43673 } } peers { permanent_uuid: "82f1641f4cb34d93a53010da960b9cc9" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 39893 } }
I20260929 20:22:35.984809 19891 raft_consensus.cc:374] T a524ff3c47bd4b0bbccfea6f3b0305e7 P fc610294087b4aaf8b0c4d9dd3d2d474 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:22:35.984925 19891 raft_consensus.cc:729] T a524ff3c47bd4b0bbccfea6f3b0305e7 P fc610294087b4aaf8b0c4d9dd3d2d474 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc610294087b4aaf8b0c4d9dd3d2d474, State: Initialized, Role: FOLLOWER
I20260929 20:22:35.985173 19891 consensus_queue.cc:260] T a524ff3c47bd4b0bbccfea6f3b0305e7 P fc610294087b4aaf8b0c4d9dd3d2d474 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba9328e853e140d6b28c75c85f061c69" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 43187 } } peers { permanent_uuid: "fc610294087b4aaf8b0c4d9dd3d2d474" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 43673 } } peers { permanent_uuid: "82f1641f4cb34d93a53010da960b9cc9" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 39893 } }
I20260929 20:22:35.985788 19891 ts_tablet_manager.cc:1428] T a524ff3c47bd4b0bbccfea6f3b0305e7 P fc610294087b4aaf8b0c4d9dd3d2d474: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260929 20:22:35.985608 19895 tablet_replica.cc:439] T a524ff3c47bd4b0bbccfea6f3b0305e7 P fc610294087b4aaf8b0c4d9dd3d2d474: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba9328e853e140d6b28c75c85f061c69" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 43187 } } peers { permanent_uuid: "fc610294087b4aaf8b0c4d9dd3d2d474" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 43673 } } peers { permanent_uuid: "82f1641f4cb34d93a53010da960b9cc9" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 39893 } } }
I20260929 20:22:35.986012 19895 tablet_replica.cc:442] T a524ff3c47bd4b0bbccfea6f3b0305e7 P fc610294087b4aaf8b0c4d9dd3d2d474: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 20:22:36.088294 19896 raft_consensus.cc:482] T a524ff3c47bd4b0bbccfea6f3b0305e7 P ba9328e853e140d6b28c75c85f061c69 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 20:22:36.088563 19896 raft_consensus.cc:504] T a524ff3c47bd4b0bbccfea6f3b0305e7 P ba9328e853e140d6b28c75c85f061c69 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba9328e853e140d6b28c75c85f061c69" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 43187 } } peers { permanent_uuid: "fc610294087b4aaf8b0c4d9dd3d2d474" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 43673 } } peers { permanent_uuid: "82f1641f4cb34d93a53010da960b9cc9" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 39893 } }
I20260929 20:22:36.089314 19896 leader_election.cc:288] T a524ff3c47bd4b0bbccfea6f3b0305e7 P ba9328e853e140d6b28c75c85f061c69 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fc610294087b4aaf8b0c4d9dd3d2d474 (127.17.53.66:43673), 82f1641f4cb34d93a53010da960b9cc9 (127.17.53.67:39893)
I20260929 20:22:36.089999 19683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a524ff3c47bd4b0bbccfea6f3b0305e7" candidate_uuid: "ba9328e853e140d6b28c75c85f061c69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc610294087b4aaf8b0c4d9dd3d2d474" is_pre_election: true
I20260929 20:22:36.089999 19819 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a524ff3c47bd4b0bbccfea6f3b0305e7" candidate_uuid: "ba9328e853e140d6b28c75c85f061c69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82f1641f4cb34d93a53010da960b9cc9" is_pre_election: true
I20260929 20:22:36.090270 19683 raft_consensus.cc:2454] T a524ff3c47bd4b0bbccfea6f3b0305e7 P fc610294087b4aaf8b0c4d9dd3d2d474 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ba9328e853e140d6b28c75c85f061c69 in term 0.
I20260929 20:22:36.090286 19819 raft_consensus.cc:2454] T a524ff3c47bd4b0bbccfea6f3b0305e7 P 82f1641f4cb34d93a53010da960b9cc9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ba9328e853e140d6b28c75c85f061c69 in term 0.
I20260929 20:22:36.090715 19469 leader_election.cc:302] T a524ff3c47bd4b0bbccfea6f3b0305e7 P ba9328e853e140d6b28c75c85f061c69 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 82f1641f4cb34d93a53010da960b9cc9, ba9328e853e140d6b28c75c85f061c69, fc610294087b4aaf8b0c4d9dd3d2d474; no voters:
I20260929 20:22:36.091171 19896 raft_consensus.cc:2781] T a524ff3c47bd4b0bbccfea6f3b0305e7 P ba9328e853e140d6b28c75c85f061c69 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 20:22:36.091262 19896 raft_consensus.cc:482] T a524ff3c47bd4b0bbccfea6f3b0305e7 P ba9328e853e140d6b28c75c85f061c69 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 20:22:36.091519 19896 raft_consensus.cc:3037] T a524ff3c47bd4b0bbccfea6f3b0305e7 P ba9328e853e140d6b28c75c85f061c69 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:22:36.093748 19896 raft_consensus.cc:504] T a524ff3c47bd4b0bbccfea6f3b0305e7 P ba9328e853e140d6b28c75c85f061c69 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba9328e853e140d6b28c75c85f061c69" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 43187 } } peers { permanent_uuid: "fc610294087b4aaf8b0c4d9dd3d2d474" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 43673 } } peers { permanent_uuid: "82f1641f4cb34d93a53010da960b9cc9" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 39893 } }
I20260929 20:22:36.094281 19896 leader_election.cc:288] T a524ff3c47bd4b0bbccfea6f3b0305e7 P ba9328e853e140d6b28c75c85f061c69 [CANDIDATE]: Term 1 election: Requested vote from peers fc610294087b4aaf8b0c4d9dd3d2d474 (127.17.53.66:43673), 82f1641f4cb34d93a53010da960b9cc9 (127.17.53.67:39893)
I20260929 20:22:36.094749 19819 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a524ff3c47bd4b0bbccfea6f3b0305e7" candidate_uuid: "ba9328e853e140d6b28c75c85f061c69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82f1641f4cb34d93a53010da960b9cc9"
I20260929 20:22:36.094846 19683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a524ff3c47bd4b0bbccfea6f3b0305e7" candidate_uuid: "ba9328e853e140d6b28c75c85f061c69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc610294087b4aaf8b0c4d9dd3d2d474"
I20260929 20:22:36.094991 19819 raft_consensus.cc:3037] T a524ff3c47bd4b0bbccfea6f3b0305e7 P 82f1641f4cb34d93a53010da960b9cc9 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:22:36.095175 19683 raft_consensus.cc:3037] T a524ff3c47bd4b0bbccfea6f3b0305e7 P fc610294087b4aaf8b0c4d9dd3d2d474 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:22:36.097206 19819 raft_consensus.cc:2454] T a524ff3c47bd4b0bbccfea6f3b0305e7 P 82f1641f4cb34d93a53010da960b9cc9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ba9328e853e140d6b28c75c85f061c69 in term 1.
I20260929 20:22:36.097589 19469 leader_election.cc:302] T a524ff3c47bd4b0bbccfea6f3b0305e7 P ba9328e853e140d6b28c75c85f061c69 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 82f1641f4cb34d93a53010da960b9cc9, ba9328e853e140d6b28c75c85f061c69; no voters:
I20260929 20:22:36.097636 19683 raft_consensus.cc:2454] T a524ff3c47bd4b0bbccfea6f3b0305e7 P fc610294087b4aaf8b0c4d9dd3d2d474 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ba9328e853e140d6b28c75c85f061c69 in term 1.
I20260929 20:22:36.097936 19896 raft_consensus.cc:2781] T a524ff3c47bd4b0bbccfea6f3b0305e7 P ba9328e853e140d6b28c75c85f061c69 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:22:36.098140 19896 raft_consensus.cc:686] T a524ff3c47bd4b0bbccfea6f3b0305e7 P ba9328e853e140d6b28c75c85f061c69 [term 1 LEADER]: Becoming Leader. State: Replica: ba9328e853e140d6b28c75c85f061c69, State: Running, Role: LEADER
I20260929 20:22:36.098403 19896 consensus_queue.cc:237] T a524ff3c47bd4b0bbccfea6f3b0305e7 P ba9328e853e140d6b28c75c85f061c69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba9328e853e140d6b28c75c85f061c69" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 43187 } } peers { permanent_uuid: "fc610294087b4aaf8b0c4d9dd3d2d474" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 43673 } } peers { permanent_uuid: "82f1641f4cb34d93a53010da960b9cc9" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 39893 } }
I20260929 20:22:36.100128 19900 tablet_replica.cc:439] T a524ff3c47bd4b0bbccfea6f3b0305e7 P ba9328e853e140d6b28c75c85f061c69: TxnStatusTablet state changed. Reason: New leader ba9328e853e140d6b28c75c85f061c69. Latest consensus state: current_term: 1 leader_uuid: "ba9328e853e140d6b28c75c85f061c69" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba9328e853e140d6b28c75c85f061c69" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 43187 } } peers { permanent_uuid: "fc610294087b4aaf8b0c4d9dd3d2d474" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 43673 } } peers { permanent_uuid: "82f1641f4cb34d93a53010da960b9cc9" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 39893 } } }
I20260929 20:22:36.100546 19900 tablet_replica.cc:442] T a524ff3c47bd4b0bbccfea6f3b0305e7 P ba9328e853e140d6b28c75c85f061c69: This TxnStatusTablet replica's current role is: LEADER
I20260929 20:22:36.101796 19378 catalog_manager.cc:5495] T a524ff3c47bd4b0bbccfea6f3b0305e7 P ba9328e853e140d6b28c75c85f061c69 reported cstate change: term changed from 0 to 1, leader changed from <none> to ba9328e853e140d6b28c75c85f061c69 (127.17.53.65). New cstate: current_term: 1 leader_uuid: "ba9328e853e140d6b28c75c85f061c69" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba9328e853e140d6b28c75c85f061c69" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 43187 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fc610294087b4aaf8b0c4d9dd3d2d474" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 43673 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "82f1641f4cb34d93a53010da960b9cc9" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 39893 } health_report { overall_health: UNKNOWN } } }
I20260929 20:22:36.103780 19912 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260929 20:22:36.104086 19912 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260929 20:22:36.147787 19907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:36.142078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.65:43187 (local address 127.0.0.1:57490)
0929 20:22:36.142343 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0929 20:22:36.142359 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 20:22:36.142520 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:22:36.143088 (+ 568us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:22:36.143092 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:22:36.143103 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 20:22:36.143299 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:22:36.143308 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:22:36.144312 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:22:36.144317 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:22:36.145320 (+ 1003us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:22:36.145330 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:22:36.146073 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:22:36.146078 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:22:36.146257 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:22:36.146309 (+ 52us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 20:22:36.147241 (+ 932us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 20:22:36.147261 (+ 20us) client_negotiation.cc:769] Sending connection context
0929 20:22:36.147346 (+ 85us) client_negotiation.cc:240] Negotiation successful
0929 20:22:36.147492 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":82}
I20260929 20:22:36.147778 19883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:36.142052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57490 (local address 127.17.53.65:43187)
0929 20:22:36.142274 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0929 20:22:36.142280 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:22:36.142428 (+ 148us) server_negotiation.cc:407] Connection header received
0929 20:22:36.142773 (+ 345us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:22:36.142777 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:22:36.142910 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:22:36.143072 (+ 162us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 20:22:36.143427 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:36.144171 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:36.145474 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:36.145909 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:36.146050 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:22:36.146515 (+ 465us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 20:22:36.147123 (+ 608us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 20:22:36.147286 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0929 20:22:36.147364 (+ 78us) server_negotiation.cc:299] Negotiation successful
0929 20:22:36.147446 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 20:22:36.156390 19683 raft_consensus.cc:1261] T a524ff3c47bd4b0bbccfea6f3b0305e7 P fc610294087b4aaf8b0c4d9dd3d2d474 [term 1 FOLLOWER]: Refusing update from remote peer ba9328e853e140d6b28c75c85f061c69: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:22:36.156538 19819 raft_consensus.cc:1261] T a524ff3c47bd4b0bbccfea6f3b0305e7 P 82f1641f4cb34d93a53010da960b9cc9 [term 1 FOLLOWER]: Refusing update from remote peer ba9328e853e140d6b28c75c85f061c69: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:22:36.157384 19900 consensus_queue.cc:1035] T a524ff3c47bd4b0bbccfea6f3b0305e7 P ba9328e853e140d6b28c75c85f061c69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82f1641f4cb34d93a53010da960b9cc9" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 39893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:22:36.157760 19896 consensus_queue.cc:1035] T a524ff3c47bd4b0bbccfea6f3b0305e7 P ba9328e853e140d6b28c75c85f061c69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc610294087b4aaf8b0c4d9dd3d2d474" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 43673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:22:36.168282 19897 tablet_replica.cc:439] T a524ff3c47bd4b0bbccfea6f3b0305e7 P 82f1641f4cb34d93a53010da960b9cc9: TxnStatusTablet state changed. Reason: New leader ba9328e853e140d6b28c75c85f061c69. Latest consensus state: current_term: 1 leader_uuid: "ba9328e853e140d6b28c75c85f061c69" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba9328e853e140d6b28c75c85f061c69" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 43187 } } peers { permanent_uuid: "fc610294087b4aaf8b0c4d9dd3d2d474" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 43673 } } peers { permanent_uuid: "82f1641f4cb34d93a53010da960b9cc9" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 39893 } } }
I20260929 20:22:36.168547 19897 tablet_replica.cc:442] T a524ff3c47bd4b0bbccfea6f3b0305e7 P 82f1641f4cb34d93a53010da960b9cc9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 20:22:36.182511 19900 tablet_replica.cc:439] T a524ff3c47bd4b0bbccfea6f3b0305e7 P ba9328e853e140d6b28c75c85f061c69: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ba9328e853e140d6b28c75c85f061c69" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba9328e853e140d6b28c75c85f061c69" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 43187 } } peers { permanent_uuid: "fc610294087b4aaf8b0c4d9dd3d2d474" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 43673 } } peers { permanent_uuid: "82f1641f4cb34d93a53010da960b9cc9" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 39893 } } }
I20260929 20:22:36.182838 19900 tablet_replica.cc:442] T a524ff3c47bd4b0bbccfea6f3b0305e7 P ba9328e853e140d6b28c75c85f061c69: This TxnStatusTablet replica's current role is: LEADER
I20260929 20:22:36.184657 19895 tablet_replica.cc:439] T a524ff3c47bd4b0bbccfea6f3b0305e7 P fc610294087b4aaf8b0c4d9dd3d2d474: TxnStatusTablet state changed. Reason: New leader ba9328e853e140d6b28c75c85f061c69. Latest consensus state: current_term: 1 leader_uuid: "ba9328e853e140d6b28c75c85f061c69" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba9328e853e140d6b28c75c85f061c69" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 43187 } } peers { permanent_uuid: "fc610294087b4aaf8b0c4d9dd3d2d474" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 43673 } } peers { permanent_uuid: "82f1641f4cb34d93a53010da960b9cc9" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 39893 } } }
I20260929 20:22:36.184971 19895 tablet_replica.cc:442] T a524ff3c47bd4b0bbccfea6f3b0305e7 P fc610294087b4aaf8b0c4d9dd3d2d474: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 20:22:36.187562 19896 tablet_replica.cc:439] T a524ff3c47bd4b0bbccfea6f3b0305e7 P ba9328e853e140d6b28c75c85f061c69: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ba9328e853e140d6b28c75c85f061c69" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba9328e853e140d6b28c75c85f061c69" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 43187 } } peers { permanent_uuid: "fc610294087b4aaf8b0c4d9dd3d2d474" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 43673 } } peers { permanent_uuid: "82f1641f4cb34d93a53010da960b9cc9" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 39893 } } }
I20260929 20:22:36.187929 19896 tablet_replica.cc:442] T a524ff3c47bd4b0bbccfea6f3b0305e7 P ba9328e853e140d6b28c75c85f061c69: This TxnStatusTablet replica's current role is: LEADER
I20260929 20:22:36.193104 19897 tablet_replica.cc:439] T a524ff3c47bd4b0bbccfea6f3b0305e7 P 82f1641f4cb34d93a53010da960b9cc9: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ba9328e853e140d6b28c75c85f061c69" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba9328e853e140d6b28c75c85f061c69" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 43187 } } peers { permanent_uuid: "fc610294087b4aaf8b0c4d9dd3d2d474" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 43673 } } peers { permanent_uuid: "82f1641f4cb34d93a53010da960b9cc9" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 39893 } } }
I20260929 20:22:36.193674 19897 tablet_replica.cc:442] T a524ff3c47bd4b0bbccfea6f3b0305e7 P 82f1641f4cb34d93a53010da960b9cc9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 20:22:36.195470 19895 tablet_replica.cc:439] T a524ff3c47bd4b0bbccfea6f3b0305e7 P fc610294087b4aaf8b0c4d9dd3d2d474: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ba9328e853e140d6b28c75c85f061c69" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba9328e853e140d6b28c75c85f061c69" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 43187 } } peers { permanent_uuid: "fc610294087b4aaf8b0c4d9dd3d2d474" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 43673 } } peers { permanent_uuid: "82f1641f4cb34d93a53010da960b9cc9" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 39893 } } }
I20260929 20:22:36.195716 19895 tablet_replica.cc:442] T a524ff3c47bd4b0bbccfea6f3b0305e7 P fc610294087b4aaf8b0c4d9dd3d2d474: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 20:22:36.204696 19883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:36.196292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57498 (local address 127.17.53.65:43187)
0929 20:22:36.196470 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0929 20:22:36.196476 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:22:36.199508 (+ 3032us) server_negotiation.cc:407] Connection header received
0929 20:22:36.199640 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:22:36.199646 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:22:36.199808 (+ 162us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:22:36.199903 (+ 95us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 20:22:36.200371 (+ 468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:36.201269 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:36.202776 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:36.203266 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:36.203424 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:22:36.203856 (+ 432us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 20:22:36.204042 (+ 186us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 20:22:36.204142 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0929 20:22:36.204274 (+ 132us) server_negotiation.cc:299] Negotiation successful
0929 20:22:36.204350 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 20:22:36.222659 19883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:36.216295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.53.65:46633 (local address 127.17.53.65:43187)
0929 20:22:36.216474 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0929 20:22:36.216480 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:22:36.217245 (+ 765us) server_negotiation.cc:407] Connection header received
0929 20:22:36.217389 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:22:36.217395 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:22:36.217479 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:22:36.217575 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 20:22:36.218061 (+ 486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:36.219233 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:36.220663 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:36.221339 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:36.221471 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:22:36.221882 (+ 411us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 20:22:36.222007 (+ 125us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 20:22:36.222111 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0929 20:22:36.222225 (+ 114us) server_negotiation.cc:299] Negotiation successful
0929 20:22:36.222305 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 20:22:36.222666 19927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:36.216295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.65:43187 (local address 127.17.53.65:46633)
0929 20:22:36.217144 (+ 849us) negotiation.cc:107] Waiting for socket to connect
0929 20:22:36.217160 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 20:22:36.217278 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:22:36.217592 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:22:36.217595 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:22:36.217603 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 20:22:36.217912 (+ 309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:22:36.217921 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:22:36.219365 (+ 1444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:22:36.219368 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:22:36.220533 (+ 1165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:22:36.220544 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:22:36.221513 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:22:36.221518 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:22:36.221645 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:22:36.221693 (+ 48us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 20:22:36.222107 (+ 414us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 20:22:36.222115 (+ 8us) client_negotiation.cc:769] Sending connection context
0929 20:22:36.222180 (+ 65us) client_negotiation.cc:240] Negotiation successful
0929 20:22:36.222284 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":649,"thread_start_us":480,"threads_started":1}
I20260929 20:22:36.232857 19819 raft_consensus.cc:1261] T 36302175c35d448ebbad5d77b50dd4ab P 82f1641f4cb34d93a53010da960b9cc9 [term 1 FOLLOWER]: Refusing update from remote peer ba9328e853e140d6b28c75c85f061c69: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:22:36.232973 19683 raft_consensus.cc:1261] T 36302175c35d448ebbad5d77b50dd4ab P fc610294087b4aaf8b0c4d9dd3d2d474 [term 1 FOLLOWER]: Refusing update from remote peer ba9328e853e140d6b28c75c85f061c69: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:22:36.233500 19896 consensus_queue.cc:1035] T 36302175c35d448ebbad5d77b50dd4ab P ba9328e853e140d6b28c75c85f061c69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc610294087b4aaf8b0c4d9dd3d2d474" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 43673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:22:36.233824 19900 consensus_queue.cc:1035] T 36302175c35d448ebbad5d77b50dd4ab P ba9328e853e140d6b28c75c85f061c69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82f1641f4cb34d93a53010da960b9cc9" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 39893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:22:36.239868 19915 mvcc.cc:205] Tried to move back new op lower bound from 7334761907124895744 to 7334761905592172544. Current Snapshot: MvccSnapshot[applied={T|T < 7334761907124895744}]
I20260929 20:22:36.293646 19379 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60614:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 20:22:36.293936 19379 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60614:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 20:22:36.297647 19379 catalog_manager.cc:5782] T 00000000000000000000000000000000 P dd8eef272ec6403e935a5cbc0ef50dce: Sending DeleteTablet for 3 replicas of tablet 36302175c35d448ebbad5d77b50dd4ab
I20260929 20:22:36.298723 19517 tablet_service.cc:1514] Processing DeleteTablet for tablet 36302175c35d448ebbad5d77b50dd4ab with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:22:36 UTC) from {username='kudu', principal='kudu/127.17.53.126@KRBTEST.COM'} at 127.0.0.1:57486
I20260929 20:22:36.298820 19658 tablet_service.cc:1514] Processing DeleteTablet for tablet 36302175c35d448ebbad5d77b50dd4ab with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:22:36 UTC) from {username='kudu', principal='kudu/127.17.53.126@KRBTEST.COM'} at 127.0.0.1:36066
I20260929 20:22:36.299144 19798 tablet_service.cc:1514] Processing DeleteTablet for tablet 36302175c35d448ebbad5d77b50dd4ab with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:22:36 UTC) from {username='kudu', principal='kudu/127.17.53.126@KRBTEST.COM'} at 127.0.0.1:50368
I20260929 20:22:36.299983 19935 tablet_replica.cc:330] T 36302175c35d448ebbad5d77b50dd4ab P 82f1641f4cb34d93a53010da960b9cc9: stopping tablet replica
I20260929 20:22:36.301009 19379 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60614:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 20:22:36.302285 19379 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 20:22:36.306236 19936 tablet_replica.cc:330] T 36302175c35d448ebbad5d77b50dd4ab P ba9328e853e140d6b28c75c85f061c69: stopping tablet replica
I20260929 20:22:36.306915 19935 raft_consensus.cc:2229] T 36302175c35d448ebbad5d77b50dd4ab P 82f1641f4cb34d93a53010da960b9cc9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 20:22:36.307336 19935 raft_consensus.cc:2258] T 36302175c35d448ebbad5d77b50dd4ab P 82f1641f4cb34d93a53010da960b9cc9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:22:36.309007 19936 raft_consensus.cc:2229] T 36302175c35d448ebbad5d77b50dd4ab P ba9328e853e140d6b28c75c85f061c69 [term 1 LEADER]: Raft consensus shutting down.
I20260929 20:22:36.309707 19936 raft_consensus.cc:2258] T 36302175c35d448ebbad5d77b50dd4ab P ba9328e853e140d6b28c75c85f061c69 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:22:36.311993 19657 tablet_service.cc:1467] Processing CreateTablet for tablet 0c5dfb57c9d740c5931931d2f938bcd7 (DEFAULT_TABLE table=test-table [id=afd39f2a96864f83b94f3f12aab08740]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:22:36.312248 19798 tablet_service.cc:1467] Processing CreateTablet for tablet 0c5dfb57c9d740c5931931d2f938bcd7 (DEFAULT_TABLE table=test-table [id=afd39f2a96864f83b94f3f12aab08740]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:22:36.312570 19657 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c5dfb57c9d740c5931931d2f938bcd7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:22:36.312666 19798 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c5dfb57c9d740c5931931d2f938bcd7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:22:36.314945 19934 tablet_replica.cc:330] T 36302175c35d448ebbad5d77b50dd4ab P fc610294087b4aaf8b0c4d9dd3d2d474: stopping tablet replica
I20260929 20:22:36.315387 19934 raft_consensus.cc:2229] T 36302175c35d448ebbad5d77b50dd4ab P fc610294087b4aaf8b0c4d9dd3d2d474 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 20:22:36.315488 19517 tablet_service.cc:1467] Processing CreateTablet for tablet 0c5dfb57c9d740c5931931d2f938bcd7 (DEFAULT_TABLE table=test-table [id=afd39f2a96864f83b94f3f12aab08740]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:22:36.315752 19934 raft_consensus.cc:2258] T 36302175c35d448ebbad5d77b50dd4ab P fc610294087b4aaf8b0c4d9dd3d2d474 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:22:36.315982 19517 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c5dfb57c9d740c5931931d2f938bcd7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:22:36.317811 19934 ts_tablet_manager.cc:1905] T 36302175c35d448ebbad5d77b50dd4ab P fc610294087b4aaf8b0c4d9dd3d2d474: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:22:36.322500 19889 tablet_bootstrap.cc:492] T 0c5dfb57c9d740c5931931d2f938bcd7 P 82f1641f4cb34d93a53010da960b9cc9: Bootstrap starting.
I20260929 20:22:36.324280 19889 tablet_bootstrap.cc:654] T 0c5dfb57c9d740c5931931d2f938bcd7 P 82f1641f4cb34d93a53010da960b9cc9: Neither blocks nor log segments found. Creating new log.
I20260929 20:22:36.325629 19934 ts_tablet_manager.cc:1918] T 36302175c35d448ebbad5d77b50dd4ab P fc610294087b4aaf8b0c4d9dd3d2d474: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 20:22:36.325810 19934 log.cc:1196] T 36302175c35d448ebbad5d77b50dd4ab P fc610294087b4aaf8b0c4d9dd3d2d474: Deleting WAL directory at /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790713339161492-17621-0/minicluster-data/ts-1/wal/wals/36302175c35d448ebbad5d77b50dd4ab
I20260929 20:22:36.326409 19934 ts_tablet_manager.cc:1939] T 36302175c35d448ebbad5d77b50dd4ab P fc610294087b4aaf8b0c4d9dd3d2d474: Deleting consensus metadata
I20260929 20:22:36.327586 19891 tablet_bootstrap.cc:492] T 0c5dfb57c9d740c5931931d2f938bcd7 P fc610294087b4aaf8b0c4d9dd3d2d474: Bootstrap starting.
I20260929 20:22:36.328334 19364 catalog_manager.cc:4841] TS fc610294087b4aaf8b0c4d9dd3d2d474 (127.17.53.66:43673): tablet 36302175c35d448ebbad5d77b50dd4ab (table test-table [id=800b83317dd04ad4ad9dab9580e5c14d]) successfully deleted
I20260929 20:22:36.329510 19891 tablet_bootstrap.cc:654] T 0c5dfb57c9d740c5931931d2f938bcd7 P fc610294087b4aaf8b0c4d9dd3d2d474: Neither blocks nor log segments found. Creating new log.
I20260929 20:22:36.331714 19891 tablet_bootstrap.cc:492] T 0c5dfb57c9d740c5931931d2f938bcd7 P fc610294087b4aaf8b0c4d9dd3d2d474: No bootstrap required, opened a new log
I20260929 20:22:36.331691 19936 ts_tablet_manager.cc:1905] T 36302175c35d448ebbad5d77b50dd4ab P ba9328e853e140d6b28c75c85f061c69: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:22:36.331975 19891 ts_tablet_manager.cc:1397] T 0c5dfb57c9d740c5931931d2f938bcd7 P fc610294087b4aaf8b0c4d9dd3d2d474: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.002s
I20260929 20:22:36.332484 19890 tablet_bootstrap.cc:492] T 0c5dfb57c9d740c5931931d2f938bcd7 P ba9328e853e140d6b28c75c85f061c69: Bootstrap starting.
I20260929 20:22:36.333006 19935 ts_tablet_manager.cc:1905] T 36302175c35d448ebbad5d77b50dd4ab P 82f1641f4cb34d93a53010da960b9cc9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:22:36.332957 19891 raft_consensus.cc:348] T 0c5dfb57c9d740c5931931d2f938bcd7 P fc610294087b4aaf8b0c4d9dd3d2d474 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba9328e853e140d6b28c75c85f061c69" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 43187 } } peers { permanent_uuid: "fc610294087b4aaf8b0c4d9dd3d2d474" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 43673 } } peers { permanent_uuid: "82f1641f4cb34d93a53010da960b9cc9" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 39893 } }
I20260929 20:22:36.333441 19891 raft_consensus.cc:374] T 0c5dfb57c9d740c5931931d2f938bcd7 P fc610294087b4aaf8b0c4d9dd3d2d474 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:22:36.333560 19891 raft_consensus.cc:729] T 0c5dfb57c9d740c5931931d2f938bcd7 P fc610294087b4aaf8b0c4d9dd3d2d474 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc610294087b4aaf8b0c4d9dd3d2d474, State: Initialized, Role: FOLLOWER
I20260929 20:22:36.333897 19891 consensus_queue.cc:260] T 0c5dfb57c9d740c5931931d2f938bcd7 P fc610294087b4aaf8b0c4d9dd3d2d474 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba9328e853e140d6b28c75c85f061c69" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 43187 } } peers { permanent_uuid: "fc610294087b4aaf8b0c4d9dd3d2d474" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 43673 } } peers { permanent_uuid: "82f1641f4cb34d93a53010da960b9cc9" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 39893 } }
I20260929 20:22:36.334296 19890 tablet_bootstrap.cc:654] T 0c5dfb57c9d740c5931931d2f938bcd7 P ba9328e853e140d6b28c75c85f061c69: Neither blocks nor log segments found. Creating new log.
I20260929 20:22:36.334656 19891 ts_tablet_manager.cc:1428] T 0c5dfb57c9d740c5931931d2f938bcd7 P fc610294087b4aaf8b0c4d9dd3d2d474: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260929 20:22:36.341971 19889 tablet_bootstrap.cc:492] T 0c5dfb57c9d740c5931931d2f938bcd7 P 82f1641f4cb34d93a53010da960b9cc9: No bootstrap required, opened a new log
I20260929 20:22:36.342155 19889 ts_tablet_manager.cc:1397] T 0c5dfb57c9d740c5931931d2f938bcd7 P 82f1641f4cb34d93a53010da960b9cc9: Time spent bootstrapping tablet: real 0.020s user 0.003s sys 0.006s
I20260929 20:22:36.343022 19889 raft_consensus.cc:348] T 0c5dfb57c9d740c5931931d2f938bcd7 P 82f1641f4cb34d93a53010da960b9cc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba9328e853e140d6b28c75c85f061c69" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 43187 } } peers { permanent_uuid: "fc610294087b4aaf8b0c4d9dd3d2d474" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 43673 } } peers { permanent_uuid: "82f1641f4cb34d93a53010da960b9cc9" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 39893 } }
I20260929 20:22:36.343266 19889 raft_consensus.cc:374] T 0c5dfb57c9d740c5931931d2f938bcd7 P 82f1641f4cb34d93a53010da960b9cc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:22:36.343362 19889 raft_consensus.cc:729] T 0c5dfb57c9d740c5931931d2f938bcd7 P 82f1641f4cb34d93a53010da960b9cc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82f1641f4cb34d93a53010da960b9cc9, State: Initialized, Role: FOLLOWER
I20260929 20:22:36.343679 19889 consensus_queue.cc:260] T 0c5dfb57c9d740c5931931d2f938bcd7 P 82f1641f4cb34d93a53010da960b9cc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba9328e853e140d6b28c75c85f061c69" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 43187 } } peers { permanent_uuid: "fc610294087b4aaf8b0c4d9dd3d2d474" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 43673 } } peers { permanent_uuid: "82f1641f4cb34d93a53010da960b9cc9" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 39893 } }
I20260929 20:22:36.346700 19890 tablet_bootstrap.cc:492] T 0c5dfb57c9d740c5931931d2f938bcd7 P ba9328e853e140d6b28c75c85f061c69: No bootstrap required, opened a new log
I20260929 20:22:36.347054 19890 ts_tablet_manager.cc:1397] T 0c5dfb57c9d740c5931931d2f938bcd7 P ba9328e853e140d6b28c75c85f061c69: Time spent bootstrapping tablet: real 0.015s user 0.003s sys 0.007s
I20260929 20:22:36.347347 19889 ts_tablet_manager.cc:1428] T 0c5dfb57c9d740c5931931d2f938bcd7 P 82f1641f4cb34d93a53010da960b9cc9: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20260929 20:22:36.347958 19890 raft_consensus.cc:348] T 0c5dfb57c9d740c5931931d2f938bcd7 P ba9328e853e140d6b28c75c85f061c69 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba9328e853e140d6b28c75c85f061c69" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 43187 } } peers { permanent_uuid: "fc610294087b4aaf8b0c4d9dd3d2d474" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 43673 } } peers { permanent_uuid: "82f1641f4cb34d93a53010da960b9cc9" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 39893 } }
I20260929 20:22:36.348268 19936 ts_tablet_manager.cc:1918] T 36302175c35d448ebbad5d77b50dd4ab P ba9328e853e140d6b28c75c85f061c69: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 20:22:36.348292 19890 raft_consensus.cc:374] T 0c5dfb57c9d740c5931931d2f938bcd7 P ba9328e853e140d6b28c75c85f061c69 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:22:36.348474 19936 log.cc:1196] T 36302175c35d448ebbad5d77b50dd4ab P ba9328e853e140d6b28c75c85f061c69: Deleting WAL directory at /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790713339161492-17621-0/minicluster-data/ts-0/wal/wals/36302175c35d448ebbad5d77b50dd4ab
I20260929 20:22:36.348587 19890 raft_consensus.cc:729] T 0c5dfb57c9d740c5931931d2f938bcd7 P ba9328e853e140d6b28c75c85f061c69 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba9328e853e140d6b28c75c85f061c69, State: Initialized, Role: FOLLOWER
I20260929 20:22:36.348927 19890 consensus_queue.cc:260] T 0c5dfb57c9d740c5931931d2f938bcd7 P ba9328e853e140d6b28c75c85f061c69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba9328e853e140d6b28c75c85f061c69" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 43187 } } peers { permanent_uuid: "fc610294087b4aaf8b0c4d9dd3d2d474" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 43673 } } peers { permanent_uuid: "82f1641f4cb34d93a53010da960b9cc9" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 39893 } }
I20260929 20:22:36.349584 19890 ts_tablet_manager.cc:1428] T 0c5dfb57c9d740c5931931d2f938bcd7 P ba9328e853e140d6b28c75c85f061c69: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260929 20:22:36.349125 19936 ts_tablet_manager.cc:1939] T 36302175c35d448ebbad5d77b50dd4ab P ba9328e853e140d6b28c75c85f061c69: Deleting consensus metadata
I20260929 20:22:36.352394 19366 catalog_manager.cc:4841] TS ba9328e853e140d6b28c75c85f061c69 (127.17.53.65:43187): tablet 36302175c35d448ebbad5d77b50dd4ab (table test-table [id=800b83317dd04ad4ad9dab9580e5c14d]) successfully deleted
I20260929 20:22:36.355156 19935 ts_tablet_manager.cc:1918] T 36302175c35d448ebbad5d77b50dd4ab P 82f1641f4cb34d93a53010da960b9cc9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 20:22:36.355322 19935 log.cc:1196] T 36302175c35d448ebbad5d77b50dd4ab P 82f1641f4cb34d93a53010da960b9cc9: Deleting WAL directory at /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790713339161492-17621-0/minicluster-data/ts-2/wal/wals/36302175c35d448ebbad5d77b50dd4ab
I20260929 20:22:36.355742 19935 ts_tablet_manager.cc:1939] T 36302175c35d448ebbad5d77b50dd4ab P 82f1641f4cb34d93a53010da960b9cc9: Deleting consensus metadata
I20260929 20:22:36.357106 19366 catalog_manager.cc:4841] TS 82f1641f4cb34d93a53010da960b9cc9 (127.17.53.67:39893): tablet 36302175c35d448ebbad5d77b50dd4ab (table test-table [id=800b83317dd04ad4ad9dab9580e5c14d]) successfully deleted
I20260929 20:22:36.360210 19895 raft_consensus.cc:482] T 0c5dfb57c9d740c5931931d2f938bcd7 P fc610294087b4aaf8b0c4d9dd3d2d474 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 20:22:36.360427 19895 raft_consensus.cc:504] T 0c5dfb57c9d740c5931931d2f938bcd7 P fc610294087b4aaf8b0c4d9dd3d2d474 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba9328e853e140d6b28c75c85f061c69" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 43187 } } peers { permanent_uuid: "fc610294087b4aaf8b0c4d9dd3d2d474" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 43673 } } peers { permanent_uuid: "82f1641f4cb34d93a53010da960b9cc9" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 39893 } }
I20260929 20:22:36.362015 19895 leader_election.cc:288] T 0c5dfb57c9d740c5931931d2f938bcd7 P fc610294087b4aaf8b0c4d9dd3d2d474 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ba9328e853e140d6b28c75c85f061c69 (127.17.53.65:43187), 82f1641f4cb34d93a53010da960b9cc9 (127.17.53.67:39893)
I20260929 20:22:36.369349 19940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:36.362183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.65:43187 (local address 127.17.53.66:46825)
0929 20:22:36.362637 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0929 20:22:36.362666 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0929 20:22:36.362799 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:22:36.363229 (+ 430us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:22:36.363234 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:22:36.363261 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:22:36.363553 (+ 292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:22:36.363565 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:22:36.365129 (+ 1564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:22:36.365136 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:22:36.367156 (+ 2020us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:22:36.367170 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:22:36.368573 (+ 1403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:22:36.368579 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:22:36.368766 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:22:36.368801 (+ 35us) client_negotiation.cc:769] Sending connection context
0929 20:22:36.368900 (+ 99us) client_negotiation.cc:240] Negotiation successful
0929 20:22:36.369024 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":333,"thread_start_us":434,"threads_started":1}
I20260929 20:22:36.369359 19883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:36.362353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.53.66:46825 (local address 127.17.53.65:43187)
0929 20:22:36.362529 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0929 20:22:36.362535 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:22:36.362752 (+ 217us) server_negotiation.cc:407] Connection header received
0929 20:22:36.362996 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:22:36.363002 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:22:36.363084 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:22:36.363185 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:22:36.363783 (+ 598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:36.364939 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:36.367316 (+ 2377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:36.368423 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:36.368591 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:22:36.368670 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0929 20:22:36.368940 (+ 270us) server_negotiation.cc:299] Negotiation successful
0929 20:22:36.369068 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 20:22:36.370543 19538 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c5dfb57c9d740c5931931d2f938bcd7" candidate_uuid: "fc610294087b4aaf8b0c4d9dd3d2d474" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba9328e853e140d6b28c75c85f061c69" is_pre_election: true
I20260929 20:22:36.371052 19538 raft_consensus.cc:2454] T 0c5dfb57c9d740c5931931d2f938bcd7 P ba9328e853e140d6b28c75c85f061c69 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fc610294087b4aaf8b0c4d9dd3d2d474 in term 0.
I20260929 20:22:36.371748 19610 leader_election.cc:302] T 0c5dfb57c9d740c5931931d2f938bcd7 P fc610294087b4aaf8b0c4d9dd3d2d474 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ba9328e853e140d6b28c75c85f061c69, fc610294087b4aaf8b0c4d9dd3d2d474; no voters:
I20260929 20:22:36.372159 19895 raft_consensus.cc:2781] T 0c5dfb57c9d740c5931931d2f938bcd7 P fc610294087b4aaf8b0c4d9dd3d2d474 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 20:22:36.372290 19895 raft_consensus.cc:482] T 0c5dfb57c9d740c5931931d2f938bcd7 P fc610294087b4aaf8b0c4d9dd3d2d474 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 20:22:36.372378 19895 raft_consensus.cc:3037] T 0c5dfb57c9d740c5931931d2f938bcd7 P fc610294087b4aaf8b0c4d9dd3d2d474 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:22:36.374293 19941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:36.363038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.67:39893 (local address 127.17.53.66:53907)
0929 20:22:36.363873 (+ 835us) negotiation.cc:107] Waiting for socket to connect
0929 20:22:36.363897 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0929 20:22:36.364063 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:22:36.367648 (+ 3585us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:22:36.367652 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:22:36.367682 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:22:36.368068 (+ 386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:22:36.368078 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:22:36.369412 (+ 1334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:22:36.369417 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:22:36.371711 (+ 2294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:22:36.371729 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:22:36.373649 (+ 1920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:22:36.373656 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:22:36.373854 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:22:36.373886 (+ 32us) client_negotiation.cc:769] Sending connection context
0929 20:22:36.373947 (+ 61us) client_negotiation.cc:240] Negotiation successful
0929 20:22:36.374041 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":663,"thread_start_us":474,"threads_started":1}
I20260929 20:22:36.375047 19942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:36.363364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.53.66:53907 (local address 127.17.53.67:39893)
0929 20:22:36.367292 (+ 3928us) server_negotiation.cc:206] Beginning negotiation
0929 20:22:36.367298 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:22:36.367314 (+ 16us) server_negotiation.cc:407] Connection header received
0929 20:22:36.367372 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:22:36.367377 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:22:36.367455 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:22:36.367595 (+ 140us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:22:36.368226 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:36.369236 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:36.371937 (+ 2701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:36.373386 (+ 1449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:36.374561 (+ 1175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:22:36.374618 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0929 20:22:36.374657 (+ 39us) server_negotiation.cc:299] Negotiation successful
0929 20:22:36.374744 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3772,"thread_start_us":3678,"threads_started":1}
I20260929 20:22:36.375582 19819 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c5dfb57c9d740c5931931d2f938bcd7" candidate_uuid: "fc610294087b4aaf8b0c4d9dd3d2d474" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82f1641f4cb34d93a53010da960b9cc9" is_pre_election: true
I20260929 20:22:36.375633 19895 raft_consensus.cc:504] T 0c5dfb57c9d740c5931931d2f938bcd7 P fc610294087b4aaf8b0c4d9dd3d2d474 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba9328e853e140d6b28c75c85f061c69" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 43187 } } peers { permanent_uuid: "fc610294087b4aaf8b0c4d9dd3d2d474" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 43673 } } peers { permanent_uuid: "82f1641f4cb34d93a53010da960b9cc9" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 39893 } }
I20260929 20:22:36.375806 19819 raft_consensus.cc:2454] T 0c5dfb57c9d740c5931931d2f938bcd7 P 82f1641f4cb34d93a53010da960b9cc9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fc610294087b4aaf8b0c4d9dd3d2d474 in term 0.
I20260929 20:22:36.376250 19895 leader_election.cc:288] T 0c5dfb57c9d740c5931931d2f938bcd7 P fc610294087b4aaf8b0c4d9dd3d2d474 [CANDIDATE]: Term 1 election: Requested vote from peers ba9328e853e140d6b28c75c85f061c69 (127.17.53.65:43187), 82f1641f4cb34d93a53010da960b9cc9 (127.17.53.67:39893)
I20260929 20:22:36.376928 19819 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c5dfb57c9d740c5931931d2f938bcd7" candidate_uuid: "fc610294087b4aaf8b0c4d9dd3d2d474" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "82f1641f4cb34d93a53010da960b9cc9"
I20260929 20:22:36.377110 19819 raft_consensus.cc:3037] T 0c5dfb57c9d740c5931931d2f938bcd7 P 82f1641f4cb34d93a53010da960b9cc9 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:22:36.379097 19819 raft_consensus.cc:2454] T 0c5dfb57c9d740c5931931d2f938bcd7 P 82f1641f4cb34d93a53010da960b9cc9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fc610294087b4aaf8b0c4d9dd3d2d474 in term 1.
I20260929 20:22:36.379076 19538 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c5dfb57c9d740c5931931d2f938bcd7" candidate_uuid: "fc610294087b4aaf8b0c4d9dd3d2d474" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba9328e853e140d6b28c75c85f061c69"
I20260929 20:22:36.379302 19538 raft_consensus.cc:3037] T 0c5dfb57c9d740c5931931d2f938bcd7 P ba9328e853e140d6b28c75c85f061c69 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:22:36.379441 19610 leader_election.cc:302] T 0c5dfb57c9d740c5931931d2f938bcd7 P fc610294087b4aaf8b0c4d9dd3d2d474 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 82f1641f4cb34d93a53010da960b9cc9, fc610294087b4aaf8b0c4d9dd3d2d474; no voters:
I20260929 20:22:36.379875 19895 raft_consensus.cc:2781] T 0c5dfb57c9d740c5931931d2f938bcd7 P fc610294087b4aaf8b0c4d9dd3d2d474 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:22:36.381826 19895 raft_consensus.cc:686] T 0c5dfb57c9d740c5931931d2f938bcd7 P fc610294087b4aaf8b0c4d9dd3d2d474 [term 1 LEADER]: Becoming Leader. State: Replica: fc610294087b4aaf8b0c4d9dd3d2d474, State: Running, Role: LEADER
I20260929 20:22:36.382251 19895 consensus_queue.cc:237] T 0c5dfb57c9d740c5931931d2f938bcd7 P fc610294087b4aaf8b0c4d9dd3d2d474 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba9328e853e140d6b28c75c85f061c69" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 43187 } } peers { permanent_uuid: "fc610294087b4aaf8b0c4d9dd3d2d474" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 43673 } } peers { permanent_uuid: "82f1641f4cb34d93a53010da960b9cc9" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 39893 } }
I20260929 20:22:36.382438 19538 raft_consensus.cc:2454] T 0c5dfb57c9d740c5931931d2f938bcd7 P ba9328e853e140d6b28c75c85f061c69 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fc610294087b4aaf8b0c4d9dd3d2d474 in term 1.
I20260929 20:22:36.386251 19379 catalog_manager.cc:5495] T 0c5dfb57c9d740c5931931d2f938bcd7 P fc610294087b4aaf8b0c4d9dd3d2d474 reported cstate change: term changed from 0 to 1, leader changed from <none> to fc610294087b4aaf8b0c4d9dd3d2d474 (127.17.53.66). New cstate: current_term: 1 leader_uuid: "fc610294087b4aaf8b0c4d9dd3d2d474" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba9328e853e140d6b28c75c85f061c69" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 43187 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fc610294087b4aaf8b0c4d9dd3d2d474" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 43673 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "82f1641f4cb34d93a53010da960b9cc9" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 39893 } health_report { overall_health: UNKNOWN } } }
I20260929 20:22:36.421868 19945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:36.415863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36078 (local address 127.17.53.66:43673)
0929 20:22:36.416581 (+ 718us) server_negotiation.cc:206] Beginning negotiation
0929 20:22:36.416587 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:22:36.416610 (+ 23us) server_negotiation.cc:407] Connection header received
0929 20:22:36.416686 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:22:36.416689 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:22:36.416765 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:22:36.416934 (+ 169us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 20:22:36.417359 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:36.418141 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:36.419576 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:36.420003 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:36.420145 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:22:36.420590 (+ 445us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 20:22:36.421174 (+ 584us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 20:22:36.421335 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0929 20:22:36.421438 (+ 103us) server_negotiation.cc:299] Negotiation successful
0929 20:22:36.421518 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":605,"thread_start_us":455,"threads_started":1}
I20260929 20:22:36.438706 19948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:36.432477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.65:43187 (local address 127.17.53.66:41931)
0929 20:22:36.433116 (+ 639us) negotiation.cc:107] Waiting for socket to connect
0929 20:22:36.433131 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 20:22:36.433317 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:22:36.433735 (+ 418us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:22:36.433740 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:22:36.433759 (+ 19us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 20:22:36.434050 (+ 291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:22:36.434059 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:22:36.435196 (+ 1137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:22:36.435202 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:22:36.436314 (+ 1112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:22:36.436326 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:22:36.437238 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:22:36.437245 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:22:36.437441 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:22:36.437498 (+ 57us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 20:22:36.438048 (+ 550us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 20:22:36.438058 (+ 10us) client_negotiation.cc:769] Sending connection context
0929 20:22:36.438141 (+ 83us) client_negotiation.cc:240] Negotiation successful
0929 20:22:36.438299 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":506,"thread_start_us":375,"threads_started":1}
I20260929 20:22:36.439248 19883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:36.432533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.53.66:41931 (local address 127.17.53.65:43187)
0929 20:22:36.432763 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0929 20:22:36.432772 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 20:22:36.433220 (+ 448us) server_negotiation.cc:407] Connection header received
0929 20:22:36.433471 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:22:36.433478 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:22:36.433602 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:22:36.433710 (+ 108us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 20:22:36.434196 (+ 486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:36.435060 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:36.436483 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:36.437067 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:36.437204 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:22:36.437729 (+ 525us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 20:22:36.437892 (+ 163us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 20:22:36.437966 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0929 20:22:36.438232 (+ 266us) server_negotiation.cc:299] Negotiation successful
0929 20:22:36.438418 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260929 20:22:36.448632 19819 raft_consensus.cc:1261] T 0c5dfb57c9d740c5931931d2f938bcd7 P 82f1641f4cb34d93a53010da960b9cc9 [term 1 FOLLOWER]: Refusing update from remote peer fc610294087b4aaf8b0c4d9dd3d2d474: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:22:36.449088 19538 raft_consensus.cc:1261] T 0c5dfb57c9d740c5931931d2f938bcd7 P ba9328e853e140d6b28c75c85f061c69 [term 1 FOLLOWER]: Refusing update from remote peer fc610294087b4aaf8b0c4d9dd3d2d474: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:22:36.449435 19943 consensus_queue.cc:1035] T 0c5dfb57c9d740c5931931d2f938bcd7 P fc610294087b4aaf8b0c4d9dd3d2d474 [LEADER]: Connected to new peer: Peer: permanent_uuid: "82f1641f4cb34d93a53010da960b9cc9" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 39893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:22:36.449810 19895 consensus_queue.cc:1035] T 0c5dfb57c9d740c5931931d2f938bcd7 P fc610294087b4aaf8b0c4d9dd3d2d474 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba9328e853e140d6b28c75c85f061c69" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 43187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:22:36.456060 19917 mvcc.cc:205] Tried to move back new op lower bound from 7334761908007878656 to 7334761907746398208. Current Snapshot: MvccSnapshot[applied={T|T < 7334761908007878656}]
I20260929 20:22:36.485855 19945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:36.479891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.53.65:51997 (local address 127.17.53.66:43673)
0929 20:22:36.480050 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0929 20:22:36.480056 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:22:36.480072 (+ 16us) server_negotiation.cc:407] Connection header received
0929 20:22:36.480129 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:22:36.480133 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:22:36.480213 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:22:36.480315 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 20:22:36.480878 (+ 563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:36.481872 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:36.484176 (+ 2304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:36.484569 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:36.484733 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:22:36.485128 (+ 395us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 20:22:36.485224 (+ 96us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 20:22:36.485327 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0929 20:22:36.485472 (+ 145us) server_negotiation.cc:299] Negotiation successful
0929 20:22:36.485547 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 20:22:36.485891 19927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:36.479179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.66:43673 (local address 127.17.53.65:51997)
0929 20:22:36.479538 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0929 20:22:36.479559 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 20:22:36.479676 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:22:36.480488 (+ 812us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:22:36.480499 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:22:36.480512 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 20:22:36.480707 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:22:36.480718 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:22:36.482051 (+ 1333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:22:36.482059 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:22:36.483791 (+ 1732us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:22:36.483808 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:22:36.484723 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:22:36.484733 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:22:36.484919 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:22:36.484959 (+ 40us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 20:22:36.485329 (+ 370us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 20:22:36.485346 (+ 17us) client_negotiation.cc:769] Sending connection context
0929 20:22:36.485452 (+ 106us) client_negotiation.cc:240] Negotiation successful
0929 20:22:36.485582 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":216}
W20260929 20:22:36.487511 19917 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260929 20:22:38.116722 19379 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60614:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 20:22:38.116964 19379 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60614:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 20:22:38.121145 19379 catalog_manager.cc:5782] T 00000000000000000000000000000000 P dd8eef272ec6403e935a5cbc0ef50dce: Sending DeleteTablet for 3 replicas of tablet 0c5dfb57c9d740c5931931d2f938bcd7
I20260929 20:22:38.122030 19657 tablet_service.cc:1514] Processing DeleteTablet for tablet 0c5dfb57c9d740c5931931d2f938bcd7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:22:38 UTC) from {username='kudu', principal='kudu/127.17.53.126@KRBTEST.COM'} at 127.0.0.1:36066
I20260929 20:22:38.122052 19798 tablet_service.cc:1514] Processing DeleteTablet for tablet 0c5dfb57c9d740c5931931d2f938bcd7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:22:38 UTC) from {username='kudu', principal='kudu/127.17.53.126@KRBTEST.COM'} at 127.0.0.1:50368
I20260929 20:22:38.122099 19517 tablet_service.cc:1514] Processing DeleteTablet for tablet 0c5dfb57c9d740c5931931d2f938bcd7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:22:38 UTC) from {username='kudu', principal='kudu/127.17.53.126@KRBTEST.COM'} at 127.0.0.1:57486
I20260929 20:22:38.136451 19975 tablet_replica.cc:330] T 0c5dfb57c9d740c5931931d2f938bcd7 P fc610294087b4aaf8b0c4d9dd3d2d474: stopping tablet replica
I20260929 20:22:38.137142 19976 tablet_replica.cc:330] T 0c5dfb57c9d740c5931931d2f938bcd7 P ba9328e853e140d6b28c75c85f061c69: stopping tablet replica
I20260929 20:22:38.137259 19977 tablet_replica.cc:330] T 0c5dfb57c9d740c5931931d2f938bcd7 P 82f1641f4cb34d93a53010da960b9cc9: stopping tablet replica
I20260929 20:22:38.137377 19976 raft_consensus.cc:2229] T 0c5dfb57c9d740c5931931d2f938bcd7 P ba9328e853e140d6b28c75c85f061c69 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 20:22:38.137452 19977 raft_consensus.cc:2229] T 0c5dfb57c9d740c5931931d2f938bcd7 P 82f1641f4cb34d93a53010da960b9cc9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 20:22:38.137655 19976 raft_consensus.cc:2258] T 0c5dfb57c9d740c5931931d2f938bcd7 P ba9328e853e140d6b28c75c85f061c69 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:22:38.137684 19977 raft_consensus.cc:2258] T 0c5dfb57c9d740c5931931d2f938bcd7 P 82f1641f4cb34d93a53010da960b9cc9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:22:38.137907 19975 raft_consensus.cc:2229] T 0c5dfb57c9d740c5931931d2f938bcd7 P fc610294087b4aaf8b0c4d9dd3d2d474 [term 1 LEADER]: Raft consensus shutting down.
I20260929 20:22:38.138527 19975 raft_consensus.cc:2258] T 0c5dfb57c9d740c5931931d2f938bcd7 P fc610294087b4aaf8b0c4d9dd3d2d474 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:22:38.139112 19977 ts_tablet_manager.cc:1905] T 0c5dfb57c9d740c5931931d2f938bcd7 P 82f1641f4cb34d93a53010da960b9cc9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:22:38.139268 19976 ts_tablet_manager.cc:1905] T 0c5dfb57c9d740c5931931d2f938bcd7 P ba9328e853e140d6b28c75c85f061c69: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:22:38.139511 19975 ts_tablet_manager.cc:1905] T 0c5dfb57c9d740c5931931d2f938bcd7 P fc610294087b4aaf8b0c4d9dd3d2d474: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:22:38.146327 19977 ts_tablet_manager.cc:1918] T 0c5dfb57c9d740c5931931d2f938bcd7 P 82f1641f4cb34d93a53010da960b9cc9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260929 20:22:38.146349 19976 ts_tablet_manager.cc:1918] T 0c5dfb57c9d740c5931931d2f938bcd7 P ba9328e853e140d6b28c75c85f061c69: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260929 20:22:38.146499 19977 log.cc:1196] T 0c5dfb57c9d740c5931931d2f938bcd7 P 82f1641f4cb34d93a53010da960b9cc9: Deleting WAL directory at /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790713339161492-17621-0/minicluster-data/ts-2/wal/wals/0c5dfb57c9d740c5931931d2f938bcd7
I20260929 20:22:38.146672 19976 log.cc:1196] T 0c5dfb57c9d740c5931931d2f938bcd7 P ba9328e853e140d6b28c75c85f061c69: Deleting WAL directory at /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790713339161492-17621-0/minicluster-data/ts-0/wal/wals/0c5dfb57c9d740c5931931d2f938bcd7
I20260929 20:22:38.146984 19977 ts_tablet_manager.cc:1939] T 0c5dfb57c9d740c5931931d2f938bcd7 P 82f1641f4cb34d93a53010da960b9cc9: Deleting consensus metadata
I20260929 20:22:38.147131 19975 ts_tablet_manager.cc:1918] T 0c5dfb57c9d740c5931931d2f938bcd7 P fc610294087b4aaf8b0c4d9dd3d2d474: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260929 20:22:38.147302 19975 log.cc:1196] T 0c5dfb57c9d740c5931931d2f938bcd7 P fc610294087b4aaf8b0c4d9dd3d2d474: Deleting WAL directory at /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790713339161492-17621-0/minicluster-data/ts-1/wal/wals/0c5dfb57c9d740c5931931d2f938bcd7
I20260929 20:22:38.147486 19976 ts_tablet_manager.cc:1939] T 0c5dfb57c9d740c5931931d2f938bcd7 P ba9328e853e140d6b28c75c85f061c69: Deleting consensus metadata
I20260929 20:22:38.147765 19975 ts_tablet_manager.cc:1939] T 0c5dfb57c9d740c5931931d2f938bcd7 P fc610294087b4aaf8b0c4d9dd3d2d474: Deleting consensus metadata
I20260929 20:22:38.148279 19366 catalog_manager.cc:4841] TS 82f1641f4cb34d93a53010da960b9cc9 (127.17.53.67:39893): tablet 0c5dfb57c9d740c5931931d2f938bcd7 (table test-table [id=afd39f2a96864f83b94f3f12aab08740]) successfully deleted
I20260929 20:22:38.148661 19366 catalog_manager.cc:4841] TS ba9328e853e140d6b28c75c85f061c69 (127.17.53.65:43187): tablet 0c5dfb57c9d740c5931931d2f938bcd7 (table test-table [id=afd39f2a96864f83b94f3f12aab08740]) successfully deleted
I20260929 20:22:38.148937 19364 catalog_manager.cc:4841] TS fc610294087b4aaf8b0c4d9dd3d2d474 (127.17.53.66:43673): tablet 0c5dfb57c9d740c5931931d2f938bcd7 (table test-table [id=afd39f2a96864f83b94f3f12aab08740]) successfully deleted
Sep 29 20:22:38 dist-test-slave-zbnc krb5kdc[19334](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713358, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 29 20:22:38 dist-test-slave-zbnc krb5kdc[19334](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713358, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.17.53.126@KRBTEST.COM
I20260929 20:22:38.183089 19985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:38.169249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60632 (local address 127.17.53.126:42509)
0929 20:22:38.169839 (+ 590us) server_negotiation.cc:206] Beginning negotiation
0929 20:22:38.169843 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:22:38.169866 (+ 23us) server_negotiation.cc:407] Connection header received
0929 20:22:38.170007 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:22:38.170012 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:22:38.170082 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:22:38.170175 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:22:38.171576 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:38.172349 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:38.173541 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:38.173903 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:38.173990 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:22:38.177951 (+ 3961us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:22:38.177956 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:22:38.177959 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:22:38.177996 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:22:38.180773 (+ 2777us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:22:38.181376 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:22:38.181380 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:22:38.181382 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:22:38.181496 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:22:38.181807 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:22:38.181810 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:22:38.181812 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:22:38.182075 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:22:38.182256 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0929 20:22:38.182536 (+ 280us) server_negotiation.cc:299] Negotiation successful
0929 20:22:38.182704 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":477,"thread_start_us":304,"threads_started":1}
I20260929 20:22:38.186848 19379 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60632:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 20:22:38.187618 19379 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 20:22:38.194017 19517 tablet_service.cc:1467] Processing CreateTablet for tablet 64425e9df3034f20927f6109d1c672fd (DEFAULT_TABLE table=test-table [id=575aa87ea9b445b29228396c975e7f05]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:22:38.194165 19657 tablet_service.cc:1467] Processing CreateTablet for tablet 64425e9df3034f20927f6109d1c672fd (DEFAULT_TABLE table=test-table [id=575aa87ea9b445b29228396c975e7f05]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:22:38.194525 19517 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 64425e9df3034f20927f6109d1c672fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:22:38.194442 19798 tablet_service.cc:1467] Processing CreateTablet for tablet 64425e9df3034f20927f6109d1c672fd (DEFAULT_TABLE table=test-table [id=575aa87ea9b445b29228396c975e7f05]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:22:38.194607 19657 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 64425e9df3034f20927f6109d1c672fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:22:38.194813 19798 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 64425e9df3034f20927f6109d1c672fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:22:38.201648 19987 tablet_bootstrap.cc:492] T 64425e9df3034f20927f6109d1c672fd P fc610294087b4aaf8b0c4d9dd3d2d474: Bootstrap starting.
I20260929 20:22:38.202587 19986 tablet_bootstrap.cc:492] T 64425e9df3034f20927f6109d1c672fd P ba9328e853e140d6b28c75c85f061c69: Bootstrap starting.
I20260929 20:22:38.203380 19988 tablet_bootstrap.cc:492] T 64425e9df3034f20927f6109d1c672fd P 82f1641f4cb34d93a53010da960b9cc9: Bootstrap starting.
I20260929 20:22:38.204566 19986 tablet_bootstrap.cc:654] T 64425e9df3034f20927f6109d1c672fd P ba9328e853e140d6b28c75c85f061c69: Neither blocks nor log segments found. Creating new log.
I20260929 20:22:38.204686 19987 tablet_bootstrap.cc:654] T 64425e9df3034f20927f6109d1c672fd P fc610294087b4aaf8b0c4d9dd3d2d474: Neither blocks nor log segments found. Creating new log.
I20260929 20:22:38.205156 19988 tablet_bootstrap.cc:654] T 64425e9df3034f20927f6109d1c672fd P 82f1641f4cb34d93a53010da960b9cc9: Neither blocks nor log segments found. Creating new log.
I20260929 20:22:38.206820 19986 tablet_bootstrap.cc:492] T 64425e9df3034f20927f6109d1c672fd P ba9328e853e140d6b28c75c85f061c69: No bootstrap required, opened a new log
I20260929 20:22:38.207020 19986 ts_tablet_manager.cc:1397] T 64425e9df3034f20927f6109d1c672fd P ba9328e853e140d6b28c75c85f061c69: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.001s
I20260929 20:22:38.207809 19986 raft_consensus.cc:348] T 64425e9df3034f20927f6109d1c672fd P ba9328e853e140d6b28c75c85f061c69 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba9328e853e140d6b28c75c85f061c69" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 43187 } } peers { permanent_uuid: "fc610294087b4aaf8b0c4d9dd3d2d474" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 43673 } } peers { permanent_uuid: "82f1641f4cb34d93a53010da960b9cc9" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 39893 } }
I20260929 20:22:38.208072 19986 raft_consensus.cc:374] T 64425e9df3034f20927f6109d1c672fd P ba9328e853e140d6b28c75c85f061c69 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:22:38.208173 19986 raft_consensus.cc:729] T 64425e9df3034f20927f6109d1c672fd P ba9328e853e140d6b28c75c85f061c69 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba9328e853e140d6b28c75c85f061c69, State: Initialized, Role: FOLLOWER
I20260929 20:22:38.208415 19986 consensus_queue.cc:260] T 64425e9df3034f20927f6109d1c672fd P ba9328e853e140d6b28c75c85f061c69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba9328e853e140d6b28c75c85f061c69" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 43187 } } peers { permanent_uuid: "fc610294087b4aaf8b0c4d9dd3d2d474" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 43673 } } peers { permanent_uuid: "82f1641f4cb34d93a53010da960b9cc9" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 39893 } }
I20260929 20:22:38.209550 19986 ts_tablet_manager.cc:1428] T 64425e9df3034f20927f6109d1c672fd P ba9328e853e140d6b28c75c85f061c69: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260929 20:22:38.211122 19988 tablet_bootstrap.cc:492] T 64425e9df3034f20927f6109d1c672fd P 82f1641f4cb34d93a53010da960b9cc9: No bootstrap required, opened a new log
I20260929 20:22:38.211306 19988 ts_tablet_manager.cc:1397] T 64425e9df3034f20927f6109d1c672fd P 82f1641f4cb34d93a53010da960b9cc9: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.006s
I20260929 20:22:38.212212 19988 raft_consensus.cc:348] T 64425e9df3034f20927f6109d1c672fd P 82f1641f4cb34d93a53010da960b9cc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba9328e853e140d6b28c75c85f061c69" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 43187 } } peers { permanent_uuid: "fc610294087b4aaf8b0c4d9dd3d2d474" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 43673 } } peers { permanent_uuid: "82f1641f4cb34d93a53010da960b9cc9" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 39893 } }
I20260929 20:22:38.212482 19988 raft_consensus.cc:374] T 64425e9df3034f20927f6109d1c672fd P 82f1641f4cb34d93a53010da960b9cc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:22:38.212608 19988 raft_consensus.cc:729] T 64425e9df3034f20927f6109d1c672fd P 82f1641f4cb34d93a53010da960b9cc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82f1641f4cb34d93a53010da960b9cc9, State: Initialized, Role: FOLLOWER
I20260929 20:22:38.212922 19988 consensus_queue.cc:260] T 64425e9df3034f20927f6109d1c672fd P 82f1641f4cb34d93a53010da960b9cc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba9328e853e140d6b28c75c85f061c69" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 43187 } } peers { permanent_uuid: "fc610294087b4aaf8b0c4d9dd3d2d474" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 43673 } } peers { permanent_uuid: "82f1641f4cb34d93a53010da960b9cc9" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 39893 } }
I20260929 20:22:38.214273 19988 ts_tablet_manager.cc:1428] T 64425e9df3034f20927f6109d1c672fd P 82f1641f4cb34d93a53010da960b9cc9: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260929 20:22:38.215642 19987 tablet_bootstrap.cc:492] T 64425e9df3034f20927f6109d1c672fd P fc610294087b4aaf8b0c4d9dd3d2d474: No bootstrap required, opened a new log
I20260929 20:22:38.215829 19987 ts_tablet_manager.cc:1397] T 64425e9df3034f20927f6109d1c672fd P fc610294087b4aaf8b0c4d9dd3d2d474: Time spent bootstrapping tablet: real 0.014s user 0.007s sys 0.003s
I20260929 20:22:38.216617 19987 raft_consensus.cc:348] T 64425e9df3034f20927f6109d1c672fd P fc610294087b4aaf8b0c4d9dd3d2d474 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba9328e853e140d6b28c75c85f061c69" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 43187 } } peers { permanent_uuid: "fc610294087b4aaf8b0c4d9dd3d2d474" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 43673 } } peers { permanent_uuid: "82f1641f4cb34d93a53010da960b9cc9" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 39893 } }
I20260929 20:22:38.216854 19987 raft_consensus.cc:374] T 64425e9df3034f20927f6109d1c672fd P fc610294087b4aaf8b0c4d9dd3d2d474 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:22:38.216969 19987 raft_consensus.cc:729] T 64425e9df3034f20927f6109d1c672fd P fc610294087b4aaf8b0c4d9dd3d2d474 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc610294087b4aaf8b0c4d9dd3d2d474, State: Initialized, Role: FOLLOWER
I20260929 20:22:38.217231 19987 consensus_queue.cc:260] T 64425e9df3034f20927f6109d1c672fd P fc610294087b4aaf8b0c4d9dd3d2d474 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba9328e853e140d6b28c75c85f061c69" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 43187 } } peers { permanent_uuid: "fc610294087b4aaf8b0c4d9dd3d2d474" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 43673 } } peers { permanent_uuid: "82f1641f4cb34d93a53010da960b9cc9" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 39893 } }
I20260929 20:22:38.217814 19987 ts_tablet_manager.cc:1428] T 64425e9df3034f20927f6109d1c672fd P fc610294087b4aaf8b0c4d9dd3d2d474: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260929 20:22:38.294499 19992 raft_consensus.cc:482] T 64425e9df3034f20927f6109d1c672fd P 82f1641f4cb34d93a53010da960b9cc9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 20:22:38.294778 19992 raft_consensus.cc:504] T 64425e9df3034f20927f6109d1c672fd P 82f1641f4cb34d93a53010da960b9cc9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba9328e853e140d6b28c75c85f061c69" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 43187 } } peers { permanent_uuid: "fc610294087b4aaf8b0c4d9dd3d2d474" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 43673 } } peers { permanent_uuid: "82f1641f4cb34d93a53010da960b9cc9" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 39893 } }
I20260929 20:22:38.296535 19992 leader_election.cc:288] T 64425e9df3034f20927f6109d1c672fd P 82f1641f4cb34d93a53010da960b9cc9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ba9328e853e140d6b28c75c85f061c69 (127.17.53.65:43187), fc610294087b4aaf8b0c4d9dd3d2d474 (127.17.53.66:43673)
I20260929 20:22:38.304960 19995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:38.296834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.53.67:44261 (local address 127.17.53.65:43187)
0929 20:22:38.298436 (+ 1602us) server_negotiation.cc:206] Beginning negotiation
0929 20:22:38.298442 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:22:38.298466 (+ 24us) server_negotiation.cc:407] Connection header received
0929 20:22:38.298543 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:22:38.298549 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:22:38.298620 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:22:38.298725 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:22:38.299268 (+ 543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:38.300348 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:38.302509 (+ 2161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:38.303708 (+ 1199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:38.303969 (+ 261us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:22:38.304034 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0929 20:22:38.304415 (+ 381us) server_negotiation.cc:299] Negotiation successful
0929 20:22:38.304564 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1453,"thread_start_us":1329,"threads_started":1}
I20260929 20:22:38.304952 19993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:38.296738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.65:43187 (local address 127.17.53.67:44261)
0929 20:22:38.297467 (+ 729us) negotiation.cc:107] Waiting for socket to connect
0929 20:22:38.297503 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0929 20:22:38.297660 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:22:38.298760 (+ 1100us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:22:38.298764 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:22:38.298800 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:22:38.299087 (+ 287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:22:38.299099 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:22:38.300481 (+ 1382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:22:38.300487 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:22:38.302353 (+ 1866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:22:38.302365 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:22:38.303955 (+ 1590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:22:38.303962 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:22:38.304164 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:22:38.304200 (+ 36us) client_negotiation.cc:769] Sending connection context
0929 20:22:38.304320 (+ 120us) client_negotiation.cc:240] Negotiation successful
0929 20:22:38.304489 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":554,"mutex_wait_us":22,"thread_start_us":315,"threads_started":1}
I20260929 20:22:38.305634 19538 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64425e9df3034f20927f6109d1c672fd" candidate_uuid: "82f1641f4cb34d93a53010da960b9cc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba9328e853e140d6b28c75c85f061c69" is_pre_election: true
I20260929 20:22:38.305892 19538 raft_consensus.cc:2454] T 64425e9df3034f20927f6109d1c672fd P ba9328e853e140d6b28c75c85f061c69 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82f1641f4cb34d93a53010da960b9cc9 in term 0.
I20260929 20:22:38.305969 19994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:38.296766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.66:43673 (local address 127.17.53.67:55843)
0929 20:22:38.297790 (+ 1024us) negotiation.cc:107] Waiting for socket to connect
0929 20:22:38.297805 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 20:22:38.297896 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:22:38.299553 (+ 1657us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:22:38.299557 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:22:38.299590 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:22:38.299970 (+ 380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:22:38.299982 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:22:38.301335 (+ 1353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:22:38.301339 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:22:38.303133 (+ 1794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:22:38.303148 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:22:38.304746 (+ 1598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:22:38.304752 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:22:38.304973 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:22:38.305347 (+ 374us) client_negotiation.cc:769] Sending connection context
0929 20:22:38.305484 (+ 137us) client_negotiation.cc:240] Negotiation successful
0929 20:22:38.305613 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":867,"mutex_wait_us":58,"thread_start_us":675,"threads_started":1}
I20260929 20:22:38.305995 19996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:38.297200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.53.67:55843 (local address 127.17.53.66:43673)
0929 20:22:38.299288 (+ 2088us) server_negotiation.cc:206] Beginning negotiation
0929 20:22:38.299298 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 20:22:38.299316 (+ 18us) server_negotiation.cc:407] Connection header received
0929 20:22:38.299368 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:22:38.299371 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:22:38.299423 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:22:38.299497 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:22:38.300151 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:38.301206 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:38.303313 (+ 2107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:38.304448 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:38.304673 (+ 225us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:22:38.304745 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0929 20:22:38.305515 (+ 770us) server_negotiation.cc:299] Negotiation successful
0929 20:22:38.305652 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1948,"thread_start_us":1801,"threads_started":1}
I20260929 20:22:38.306517 19750 leader_election.cc:302] T 64425e9df3034f20927f6109d1c672fd P 82f1641f4cb34d93a53010da960b9cc9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 82f1641f4cb34d93a53010da960b9cc9, ba9328e853e140d6b28c75c85f061c69; no voters:
I20260929 20:22:38.306586 19682 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64425e9df3034f20927f6109d1c672fd" candidate_uuid: "82f1641f4cb34d93a53010da960b9cc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc610294087b4aaf8b0c4d9dd3d2d474" is_pre_election: true
I20260929 20:22:38.306814 19682 raft_consensus.cc:2454] T 64425e9df3034f20927f6109d1c672fd P fc610294087b4aaf8b0c4d9dd3d2d474 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82f1641f4cb34d93a53010da960b9cc9 in term 0.
I20260929 20:22:38.306973 19992 raft_consensus.cc:2781] T 64425e9df3034f20927f6109d1c672fd P 82f1641f4cb34d93a53010da960b9cc9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 20:22:38.307097 19992 raft_consensus.cc:482] T 64425e9df3034f20927f6109d1c672fd P 82f1641f4cb34d93a53010da960b9cc9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 20:22:38.307235 19992 raft_consensus.cc:3037] T 64425e9df3034f20927f6109d1c672fd P 82f1641f4cb34d93a53010da960b9cc9 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:22:38.309477 19992 raft_consensus.cc:504] T 64425e9df3034f20927f6109d1c672fd P 82f1641f4cb34d93a53010da960b9cc9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba9328e853e140d6b28c75c85f061c69" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 43187 } } peers { permanent_uuid: "fc610294087b4aaf8b0c4d9dd3d2d474" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 43673 } } peers { permanent_uuid: "82f1641f4cb34d93a53010da960b9cc9" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 39893 } }
I20260929 20:22:38.310107 19992 leader_election.cc:288] T 64425e9df3034f20927f6109d1c672fd P 82f1641f4cb34d93a53010da960b9cc9 [CANDIDATE]: Term 1 election: Requested vote from peers ba9328e853e140d6b28c75c85f061c69 (127.17.53.65:43187), fc610294087b4aaf8b0c4d9dd3d2d474 (127.17.53.66:43673)
I20260929 20:22:38.310513 19538 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64425e9df3034f20927f6109d1c672fd" candidate_uuid: "82f1641f4cb34d93a53010da960b9cc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba9328e853e140d6b28c75c85f061c69"
I20260929 20:22:38.310585 19682 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "64425e9df3034f20927f6109d1c672fd" candidate_uuid: "82f1641f4cb34d93a53010da960b9cc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc610294087b4aaf8b0c4d9dd3d2d474"
I20260929 20:22:38.310745 19538 raft_consensus.cc:3037] T 64425e9df3034f20927f6109d1c672fd P ba9328e853e140d6b28c75c85f061c69 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:22:38.310767 19682 raft_consensus.cc:3037] T 64425e9df3034f20927f6109d1c672fd P fc610294087b4aaf8b0c4d9dd3d2d474 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:22:38.312880 19538 raft_consensus.cc:2454] T 64425e9df3034f20927f6109d1c672fd P ba9328e853e140d6b28c75c85f061c69 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82f1641f4cb34d93a53010da960b9cc9 in term 1.
I20260929 20:22:38.312930 19682 raft_consensus.cc:2454] T 64425e9df3034f20927f6109d1c672fd P fc610294087b4aaf8b0c4d9dd3d2d474 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82f1641f4cb34d93a53010da960b9cc9 in term 1.
I20260929 20:22:38.313282 19750 leader_election.cc:302] T 64425e9df3034f20927f6109d1c672fd P 82f1641f4cb34d93a53010da960b9cc9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 82f1641f4cb34d93a53010da960b9cc9, ba9328e853e140d6b28c75c85f061c69; no voters:
I20260929 20:22:38.313565 19992 raft_consensus.cc:2781] T 64425e9df3034f20927f6109d1c672fd P 82f1641f4cb34d93a53010da960b9cc9 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:22:38.314035 19992 raft_consensus.cc:686] T 64425e9df3034f20927f6109d1c672fd P 82f1641f4cb34d93a53010da960b9cc9 [term 1 LEADER]: Becoming Leader. State: Replica: 82f1641f4cb34d93a53010da960b9cc9, State: Running, Role: LEADER
I20260929 20:22:38.314424 19992 consensus_queue.cc:237] T 64425e9df3034f20927f6109d1c672fd P 82f1641f4cb34d93a53010da960b9cc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba9328e853e140d6b28c75c85f061c69" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 43187 } } peers { permanent_uuid: "fc610294087b4aaf8b0c4d9dd3d2d474" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 43673 } } peers { permanent_uuid: "82f1641f4cb34d93a53010da960b9cc9" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 39893 } }
I20260929 20:22:38.317958 19379 catalog_manager.cc:5495] T 64425e9df3034f20927f6109d1c672fd P 82f1641f4cb34d93a53010da960b9cc9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 82f1641f4cb34d93a53010da960b9cc9 (127.17.53.67). New cstate: current_term: 1 leader_uuid: "82f1641f4cb34d93a53010da960b9cc9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba9328e853e140d6b28c75c85f061c69" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 43187 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fc610294087b4aaf8b0c4d9dd3d2d474" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 43673 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "82f1641f4cb34d93a53010da960b9cc9" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 39893 } health_report { overall_health: HEALTHY } } }
I20260929 20:22:38.368723 20000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:38.362646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50372 (local address 127.17.53.67:39893)
0929 20:22:38.363342 (+ 696us) server_negotiation.cc:206] Beginning negotiation
0929 20:22:38.363346 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:22:38.363364 (+ 18us) server_negotiation.cc:407] Connection header received
0929 20:22:38.363444 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:22:38.363448 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:22:38.363519 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:22:38.363696 (+ 177us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 20:22:38.364163 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:38.364934 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:38.366272 (+ 1338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:38.366772 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:38.366935 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:22:38.367393 (+ 458us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 20:22:38.368058 (+ 665us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 20:22:38.368211 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0929 20:22:38.368309 (+ 98us) server_negotiation.cc:299] Negotiation successful
0929 20:22:38.368396 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":563,"thread_start_us":422,"threads_started":1}
I20260929 20:22:38.382957 19995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:38.378321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.53.67:44523 (local address 127.17.53.65:43187)
0929 20:22:38.378559 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0929 20:22:38.378565 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:22:38.378806 (+ 241us) server_negotiation.cc:407] Connection header received
0929 20:22:38.379031 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:22:38.379034 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:22:38.379100 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:22:38.379174 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 20:22:38.379607 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:38.380282 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:38.381555 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:38.381918 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:38.382004 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:22:38.382357 (+ 353us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 20:22:38.382447 (+ 90us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 20:22:38.382509 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0929 20:22:38.382627 (+ 118us) server_negotiation.cc:299] Negotiation successful
0929 20:22:38.382682 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260929 20:22:38.383073 20003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:38.378115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.65:43187 (local address 127.17.53.67:44523)
0929 20:22:38.378710 (+ 595us) negotiation.cc:107] Waiting for socket to connect
0929 20:22:38.378725 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 20:22:38.378849 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:22:38.379199 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:22:38.379202 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:22:38.379210 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 20:22:38.379470 (+ 260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:22:38.379477 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:22:38.380392 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:22:38.380395 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:22:38.381442 (+ 1047us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:22:38.381453 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:22:38.382058 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:22:38.382062 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:22:38.382168 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:22:38.382217 (+ 49us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 20:22:38.382529 (+ 312us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 20:22:38.382536 (+ 7us) client_negotiation.cc:769] Sending connection context
0929 20:22:38.382595 (+ 59us) client_negotiation.cc:240] Negotiation successful
0929 20:22:38.382688 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":318,"threads_started":1}
I20260929 20:22:38.391180 19538 raft_consensus.cc:1261] T 64425e9df3034f20927f6109d1c672fd P ba9328e853e140d6b28c75c85f061c69 [term 1 FOLLOWER]: Refusing update from remote peer 82f1641f4cb34d93a53010da960b9cc9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:22:38.391708 19682 raft_consensus.cc:1261] T 64425e9df3034f20927f6109d1c672fd P fc610294087b4aaf8b0c4d9dd3d2d474 [term 1 FOLLOWER]: Refusing update from remote peer 82f1641f4cb34d93a53010da960b9cc9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:22:38.391849 19997 consensus_queue.cc:1035] T 64425e9df3034f20927f6109d1c672fd P 82f1641f4cb34d93a53010da960b9cc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba9328e853e140d6b28c75c85f061c69" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 43187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:22:38.392338 19992 consensus_queue.cc:1035] T 64425e9df3034f20927f6109d1c672fd P 82f1641f4cb34d93a53010da960b9cc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc610294087b4aaf8b0c4d9dd3d2d474" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 43673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:22:38.397763 19956 mvcc.cc:205] Tried to move back new op lower bound from 7334761915964715008 to 7334761915659665408. Current Snapshot: MvccSnapshot[applied={T|T < 7334761915964715008}]
I20260929 20:22:38.429740 20000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:38.423942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.53.65:33351 (local address 127.17.53.67:39893)
0929 20:22:38.424168 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0929 20:22:38.424175 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:22:38.424660 (+ 485us) server_negotiation.cc:407] Connection header received
0929 20:22:38.424894 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:22:38.424898 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:22:38.424960 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:22:38.425033 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 20:22:38.425483 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:38.426212 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:38.427849 (+ 1637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:38.428412 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:38.428539 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:22:38.429013 (+ 474us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 20:22:38.429148 (+ 135us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 20:22:38.429248 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0929 20:22:38.429359 (+ 111us) server_negotiation.cc:299] Negotiation successful
0929 20:22:38.429430 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260929 20:22:38.429857 20008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:38.423856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.67:39893 (local address 127.17.53.65:33351)
0929 20:22:38.424559 (+ 703us) negotiation.cc:107] Waiting for socket to connect
0929 20:22:38.424588 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0929 20:22:38.424719 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:22:38.425117 (+ 398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:22:38.425121 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:22:38.425131 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 20:22:38.425362 (+ 231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:22:38.425369 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:22:38.426332 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:22:38.426336 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:22:38.427707 (+ 1371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:22:38.427722 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:22:38.428601 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:22:38.428608 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:22:38.428799 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:22:38.428854 (+ 55us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 20:22:38.429251 (+ 397us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 20:22:38.429264 (+ 13us) client_negotiation.cc:769] Sending connection context
0929 20:22:38.429342 (+ 78us) client_negotiation.cc:240] Negotiation successful
0929 20:22:38.429479 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":536,"thread_start_us":353,"threads_started":1}
I20260929 20:22:38.462548 19378 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60632:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 20:22:38.462817 19378 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60632:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 20:22:38.465761 19378 catalog_manager.cc:5782] T 00000000000000000000000000000000 P dd8eef272ec6403e935a5cbc0ef50dce: Sending DeleteTablet for 3 replicas of tablet 64425e9df3034f20927f6109d1c672fd
I20260929 20:22:38.466676 19517 tablet_service.cc:1514] Processing DeleteTablet for tablet 64425e9df3034f20927f6109d1c672fd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:22:38 UTC) from {username='kudu', principal='kudu/127.17.53.126@KRBTEST.COM'} at 127.0.0.1:57486
I20260929 20:22:38.466820 19657 tablet_service.cc:1514] Processing DeleteTablet for tablet 64425e9df3034f20927f6109d1c672fd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:22:38 UTC) from {username='kudu', principal='kudu/127.17.53.126@KRBTEST.COM'} at 127.0.0.1:36066
I20260929 20:22:38.467046 19976 tablet_replica.cc:330] T 64425e9df3034f20927f6109d1c672fd P ba9328e853e140d6b28c75c85f061c69: stopping tablet replica
I20260929 20:22:38.467142 19975 tablet_replica.cc:330] T 64425e9df3034f20927f6109d1c672fd P fc610294087b4aaf8b0c4d9dd3d2d474: stopping tablet replica
I20260929 20:22:38.467289 19976 raft_consensus.cc:2229] T 64425e9df3034f20927f6109d1c672fd P ba9328e853e140d6b28c75c85f061c69 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 20:22:38.467386 19975 raft_consensus.cc:2229] T 64425e9df3034f20927f6109d1c672fd P fc610294087b4aaf8b0c4d9dd3d2d474 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 20:22:38.467509 19976 raft_consensus.cc:2258] T 64425e9df3034f20927f6109d1c672fd P ba9328e853e140d6b28c75c85f061c69 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:22:38.467609 19975 raft_consensus.cc:2258] T 64425e9df3034f20927f6109d1c672fd P fc610294087b4aaf8b0c4d9dd3d2d474 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:22:38.468292 19798 tablet_service.cc:1514] Processing DeleteTablet for tablet 64425e9df3034f20927f6109d1c672fd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:22:38 UTC) from {username='kudu', principal='kudu/127.17.53.126@KRBTEST.COM'} at 127.0.0.1:50368
I20260929 20:22:38.468568 19977 tablet_replica.cc:330] T 64425e9df3034f20927f6109d1c672fd P 82f1641f4cb34d93a53010da960b9cc9: stopping tablet replica
I20260929 20:22:38.468685 19976 ts_tablet_manager.cc:1905] T 64425e9df3034f20927f6109d1c672fd P ba9328e853e140d6b28c75c85f061c69: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:22:38.468791 19977 raft_consensus.cc:2229] T 64425e9df3034f20927f6109d1c672fd P 82f1641f4cb34d93a53010da960b9cc9 [term 1 LEADER]: Raft consensus shutting down.
I20260929 20:22:38.469074 19975 ts_tablet_manager.cc:1905] T 64425e9df3034f20927f6109d1c672fd P fc610294087b4aaf8b0c4d9dd3d2d474: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:22:38.469290 19977 raft_consensus.cc:2258] T 64425e9df3034f20927f6109d1c672fd P 82f1641f4cb34d93a53010da960b9cc9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:22:38.473795 19975 ts_tablet_manager.cc:1918] T 64425e9df3034f20927f6109d1c672fd P fc610294087b4aaf8b0c4d9dd3d2d474: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 20:22:38.473961 19975 log.cc:1196] T 64425e9df3034f20927f6109d1c672fd P fc610294087b4aaf8b0c4d9dd3d2d474: Deleting WAL directory at /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790713339161492-17621-0/minicluster-data/ts-1/wal/wals/64425e9df3034f20927f6109d1c672fd
I20260929 20:22:38.474423 19975 ts_tablet_manager.cc:1939] T 64425e9df3034f20927f6109d1c672fd P fc610294087b4aaf8b0c4d9dd3d2d474: Deleting consensus metadata
I20260929 20:22:38.474730 19977 ts_tablet_manager.cc:1905] T 64425e9df3034f20927f6109d1c672fd P 82f1641f4cb34d93a53010da960b9cc9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:22:38.475497 19364 catalog_manager.cc:4841] TS fc610294087b4aaf8b0c4d9dd3d2d474 (127.17.53.66:43673): tablet 64425e9df3034f20927f6109d1c672fd (table test-table [id=575aa87ea9b445b29228396c975e7f05]) successfully deleted
I20260929 20:22:38.475746 19976 ts_tablet_manager.cc:1918] T 64425e9df3034f20927f6109d1c672fd P ba9328e853e140d6b28c75c85f061c69: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 20:22:38.475886 19976 log.cc:1196] T 64425e9df3034f20927f6109d1c672fd P ba9328e853e140d6b28c75c85f061c69: Deleting WAL directory at /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790713339161492-17621-0/minicluster-data/ts-0/wal/wals/64425e9df3034f20927f6109d1c672fd
I20260929 20:22:38.476366 19976 ts_tablet_manager.cc:1939] T 64425e9df3034f20927f6109d1c672fd P ba9328e853e140d6b28c75c85f061c69: Deleting consensus metadata
I20260929 20:22:38.476642 19378 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60632:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 20:22:38.477346 19378 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 20:22:38.477852 19366 catalog_manager.cc:4841] TS ba9328e853e140d6b28c75c85f061c69 (127.17.53.65:43187): tablet 64425e9df3034f20927f6109d1c672fd (table test-table [id=575aa87ea9b445b29228396c975e7f05]) successfully deleted
I20260929 20:22:38.481388 19977 ts_tablet_manager.cc:1918] T 64425e9df3034f20927f6109d1c672fd P 82f1641f4cb34d93a53010da960b9cc9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 20:22:38.481544 19977 log.cc:1196] T 64425e9df3034f20927f6109d1c672fd P 82f1641f4cb34d93a53010da960b9cc9: Deleting WAL directory at /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790713339161492-17621-0/minicluster-data/ts-2/wal/wals/64425e9df3034f20927f6109d1c672fd
I20260929 20:22:38.481979 19977 ts_tablet_manager.cc:1939] T 64425e9df3034f20927f6109d1c672fd P 82f1641f4cb34d93a53010da960b9cc9: Deleting consensus metadata
I20260929 20:22:38.483114 19366 catalog_manager.cc:4841] TS 82f1641f4cb34d93a53010da960b9cc9 (127.17.53.67:39893): tablet 64425e9df3034f20927f6109d1c672fd (table test-table [id=575aa87ea9b445b29228396c975e7f05]) successfully deleted
I20260929 20:22:38.485255 19517 tablet_service.cc:1467] Processing CreateTablet for tablet a70a34f5477a476e89b051729b8aa92d (DEFAULT_TABLE table=test-table [id=d1131e0947344c01a4b54266196ed965]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:22:38.485747 19517 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a70a34f5477a476e89b051729b8aa92d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:22:38.485651 19657 tablet_service.cc:1467] Processing CreateTablet for tablet a70a34f5477a476e89b051729b8aa92d (DEFAULT_TABLE table=test-table [id=d1131e0947344c01a4b54266196ed965]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:22:38.486052 19657 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a70a34f5477a476e89b051729b8aa92d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:22:38.487600 19798 tablet_service.cc:1467] Processing CreateTablet for tablet a70a34f5477a476e89b051729b8aa92d (DEFAULT_TABLE table=test-table [id=d1131e0947344c01a4b54266196ed965]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:22:38.487989 19798 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a70a34f5477a476e89b051729b8aa92d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:22:38.493640 19987 tablet_bootstrap.cc:492] T a70a34f5477a476e89b051729b8aa92d P fc610294087b4aaf8b0c4d9dd3d2d474: Bootstrap starting.
I20260929 20:22:38.495332 19987 tablet_bootstrap.cc:654] T a70a34f5477a476e89b051729b8aa92d P fc610294087b4aaf8b0c4d9dd3d2d474: Neither blocks nor log segments found. Creating new log.
I20260929 20:22:38.500861 19986 tablet_bootstrap.cc:492] T a70a34f5477a476e89b051729b8aa92d P ba9328e853e140d6b28c75c85f061c69: Bootstrap starting.
I20260929 20:22:38.502137 19988 tablet_bootstrap.cc:492] T a70a34f5477a476e89b051729b8aa92d P 82f1641f4cb34d93a53010da960b9cc9: Bootstrap starting.
I20260929 20:22:38.502614 19986 tablet_bootstrap.cc:654] T a70a34f5477a476e89b051729b8aa92d P ba9328e853e140d6b28c75c85f061c69: Neither blocks nor log segments found. Creating new log.
I20260929 20:22:38.504009 19988 tablet_bootstrap.cc:654] T a70a34f5477a476e89b051729b8aa92d P 82f1641f4cb34d93a53010da960b9cc9: Neither blocks nor log segments found. Creating new log.
I20260929 20:22:38.511672 19987 tablet_bootstrap.cc:492] T a70a34f5477a476e89b051729b8aa92d P fc610294087b4aaf8b0c4d9dd3d2d474: No bootstrap required, opened a new log
I20260929 20:22:38.511881 19987 ts_tablet_manager.cc:1397] T a70a34f5477a476e89b051729b8aa92d P fc610294087b4aaf8b0c4d9dd3d2d474: Time spent bootstrapping tablet: real 0.018s user 0.009s sys 0.002s
I20260929 20:22:38.512621 19987 raft_consensus.cc:348] T a70a34f5477a476e89b051729b8aa92d P fc610294087b4aaf8b0c4d9dd3d2d474 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba9328e853e140d6b28c75c85f061c69" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 43187 } } peers { permanent_uuid: "fc610294087b4aaf8b0c4d9dd3d2d474" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 43673 } } peers { permanent_uuid: "82f1641f4cb34d93a53010da960b9cc9" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 39893 } }
I20260929 20:22:38.512863 19987 raft_consensus.cc:374] T a70a34f5477a476e89b051729b8aa92d P fc610294087b4aaf8b0c4d9dd3d2d474 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:22:38.512939 19987 raft_consensus.cc:729] T a70a34f5477a476e89b051729b8aa92d P fc610294087b4aaf8b0c4d9dd3d2d474 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc610294087b4aaf8b0c4d9dd3d2d474, State: Initialized, Role: FOLLOWER
I20260929 20:22:38.513193 19987 consensus_queue.cc:260] T a70a34f5477a476e89b051729b8aa92d P fc610294087b4aaf8b0c4d9dd3d2d474 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba9328e853e140d6b28c75c85f061c69" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 43187 } } peers { permanent_uuid: "fc610294087b4aaf8b0c4d9dd3d2d474" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 43673 } } peers { permanent_uuid: "82f1641f4cb34d93a53010da960b9cc9" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 39893 } }
I20260929 20:22:38.514334 19987 ts_tablet_manager.cc:1428] T a70a34f5477a476e89b051729b8aa92d P fc610294087b4aaf8b0c4d9dd3d2d474: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260929 20:22:38.515501 19988 tablet_bootstrap.cc:492] T a70a34f5477a476e89b051729b8aa92d P 82f1641f4cb34d93a53010da960b9cc9: No bootstrap required, opened a new log
I20260929 20:22:38.515674 19988 ts_tablet_manager.cc:1397] T a70a34f5477a476e89b051729b8aa92d P 82f1641f4cb34d93a53010da960b9cc9: Time spent bootstrapping tablet: real 0.014s user 0.000s sys 0.007s
I20260929 20:22:38.516060 19986 tablet_bootstrap.cc:492] T a70a34f5477a476e89b051729b8aa92d P ba9328e853e140d6b28c75c85f061c69: No bootstrap required, opened a new log
I20260929 20:22:38.516286 19986 ts_tablet_manager.cc:1397] T a70a34f5477a476e89b051729b8aa92d P ba9328e853e140d6b28c75c85f061c69: Time spent bootstrapping tablet: real 0.015s user 0.000s sys 0.009s
I20260929 20:22:38.516369 19988 raft_consensus.cc:348] T a70a34f5477a476e89b051729b8aa92d P 82f1641f4cb34d93a53010da960b9cc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba9328e853e140d6b28c75c85f061c69" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 43187 } } peers { permanent_uuid: "fc610294087b4aaf8b0c4d9dd3d2d474" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 43673 } } peers { permanent_uuid: "82f1641f4cb34d93a53010da960b9cc9" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 39893 } }
I20260929 20:22:38.516603 19988 raft_consensus.cc:374] T a70a34f5477a476e89b051729b8aa92d P 82f1641f4cb34d93a53010da960b9cc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:22:38.516690 19988 raft_consensus.cc:729] T a70a34f5477a476e89b051729b8aa92d P 82f1641f4cb34d93a53010da960b9cc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82f1641f4cb34d93a53010da960b9cc9, State: Initialized, Role: FOLLOWER
I20260929 20:22:38.516904 19988 consensus_queue.cc:260] T a70a34f5477a476e89b051729b8aa92d P 82f1641f4cb34d93a53010da960b9cc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba9328e853e140d6b28c75c85f061c69" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 43187 } } peers { permanent_uuid: "fc610294087b4aaf8b0c4d9dd3d2d474" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 43673 } } peers { permanent_uuid: "82f1641f4cb34d93a53010da960b9cc9" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 39893 } }
I20260929 20:22:38.517417 19988 ts_tablet_manager.cc:1428] T a70a34f5477a476e89b051729b8aa92d P 82f1641f4cb34d93a53010da960b9cc9: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260929 20:22:38.517396 19986 raft_consensus.cc:348] T a70a34f5477a476e89b051729b8aa92d P ba9328e853e140d6b28c75c85f061c69 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba9328e853e140d6b28c75c85f061c69" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 43187 } } peers { permanent_uuid: "fc610294087b4aaf8b0c4d9dd3d2d474" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 43673 } } peers { permanent_uuid: "82f1641f4cb34d93a53010da960b9cc9" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 39893 } }
I20260929 20:22:38.517798 19986 raft_consensus.cc:374] T a70a34f5477a476e89b051729b8aa92d P ba9328e853e140d6b28c75c85f061c69 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:22:38.518057 19986 raft_consensus.cc:729] T a70a34f5477a476e89b051729b8aa92d P ba9328e853e140d6b28c75c85f061c69 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba9328e853e140d6b28c75c85f061c69, State: Initialized, Role: FOLLOWER
I20260929 20:22:38.518389 19986 consensus_queue.cc:260] T a70a34f5477a476e89b051729b8aa92d P ba9328e853e140d6b28c75c85f061c69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba9328e853e140d6b28c75c85f061c69" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 43187 } } peers { permanent_uuid: "fc610294087b4aaf8b0c4d9dd3d2d474" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 43673 } } peers { permanent_uuid: "82f1641f4cb34d93a53010da960b9cc9" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 39893 } }
I20260929 20:22:38.519021 19986 ts_tablet_manager.cc:1428] T a70a34f5477a476e89b051729b8aa92d P ba9328e853e140d6b28c75c85f061c69: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20260929 20:22:38.673574 19992 raft_consensus.cc:482] T a70a34f5477a476e89b051729b8aa92d P 82f1641f4cb34d93a53010da960b9cc9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 20:22:38.673813 19992 raft_consensus.cc:504] T a70a34f5477a476e89b051729b8aa92d P 82f1641f4cb34d93a53010da960b9cc9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba9328e853e140d6b28c75c85f061c69" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 43187 } } peers { permanent_uuid: "fc610294087b4aaf8b0c4d9dd3d2d474" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 43673 } } peers { permanent_uuid: "82f1641f4cb34d93a53010da960b9cc9" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 39893 } }
I20260929 20:22:38.674484 19992 leader_election.cc:288] T a70a34f5477a476e89b051729b8aa92d P 82f1641f4cb34d93a53010da960b9cc9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ba9328e853e140d6b28c75c85f061c69 (127.17.53.65:43187), fc610294087b4aaf8b0c4d9dd3d2d474 (127.17.53.66:43673)
I20260929 20:22:38.675146 19683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a70a34f5477a476e89b051729b8aa92d" candidate_uuid: "82f1641f4cb34d93a53010da960b9cc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc610294087b4aaf8b0c4d9dd3d2d474" is_pre_election: true
I20260929 20:22:38.675166 19538 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a70a34f5477a476e89b051729b8aa92d" candidate_uuid: "82f1641f4cb34d93a53010da960b9cc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba9328e853e140d6b28c75c85f061c69" is_pre_election: true
I20260929 20:22:38.675402 19538 raft_consensus.cc:2454] T a70a34f5477a476e89b051729b8aa92d P ba9328e853e140d6b28c75c85f061c69 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82f1641f4cb34d93a53010da960b9cc9 in term 0.
I20260929 20:22:38.675403 19683 raft_consensus.cc:2454] T a70a34f5477a476e89b051729b8aa92d P fc610294087b4aaf8b0c4d9dd3d2d474 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82f1641f4cb34d93a53010da960b9cc9 in term 0.
I20260929 20:22:38.675874 19748 leader_election.cc:302] T a70a34f5477a476e89b051729b8aa92d P 82f1641f4cb34d93a53010da960b9cc9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 82f1641f4cb34d93a53010da960b9cc9, fc610294087b4aaf8b0c4d9dd3d2d474; no voters:
I20260929 20:22:38.676173 19992 raft_consensus.cc:2781] T a70a34f5477a476e89b051729b8aa92d P 82f1641f4cb34d93a53010da960b9cc9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 20:22:38.676301 19992 raft_consensus.cc:482] T a70a34f5477a476e89b051729b8aa92d P 82f1641f4cb34d93a53010da960b9cc9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 20:22:38.676424 19992 raft_consensus.cc:3037] T a70a34f5477a476e89b051729b8aa92d P 82f1641f4cb34d93a53010da960b9cc9 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:22:38.678774 19992 raft_consensus.cc:504] T a70a34f5477a476e89b051729b8aa92d P 82f1641f4cb34d93a53010da960b9cc9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba9328e853e140d6b28c75c85f061c69" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 43187 } } peers { permanent_uuid: "fc610294087b4aaf8b0c4d9dd3d2d474" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 43673 } } peers { permanent_uuid: "82f1641f4cb34d93a53010da960b9cc9" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 39893 } }
I20260929 20:22:38.679441 19992 leader_election.cc:288] T a70a34f5477a476e89b051729b8aa92d P 82f1641f4cb34d93a53010da960b9cc9 [CANDIDATE]: Term 1 election: Requested vote from peers ba9328e853e140d6b28c75c85f061c69 (127.17.53.65:43187), fc610294087b4aaf8b0c4d9dd3d2d474 (127.17.53.66:43673)
I20260929 20:22:38.680009 19538 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a70a34f5477a476e89b051729b8aa92d" candidate_uuid: "82f1641f4cb34d93a53010da960b9cc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba9328e853e140d6b28c75c85f061c69"
I20260929 20:22:38.680011 19683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a70a34f5477a476e89b051729b8aa92d" candidate_uuid: "82f1641f4cb34d93a53010da960b9cc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc610294087b4aaf8b0c4d9dd3d2d474"
I20260929 20:22:38.680236 19538 raft_consensus.cc:3037] T a70a34f5477a476e89b051729b8aa92d P ba9328e853e140d6b28c75c85f061c69 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:22:38.680233 19683 raft_consensus.cc:3037] T a70a34f5477a476e89b051729b8aa92d P fc610294087b4aaf8b0c4d9dd3d2d474 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:22:38.683305 19538 raft_consensus.cc:2454] T a70a34f5477a476e89b051729b8aa92d P ba9328e853e140d6b28c75c85f061c69 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82f1641f4cb34d93a53010da960b9cc9 in term 1.
I20260929 20:22:38.683305 19683 raft_consensus.cc:2454] T a70a34f5477a476e89b051729b8aa92d P fc610294087b4aaf8b0c4d9dd3d2d474 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82f1641f4cb34d93a53010da960b9cc9 in term 1.
I20260929 20:22:38.683779 19748 leader_election.cc:302] T a70a34f5477a476e89b051729b8aa92d P 82f1641f4cb34d93a53010da960b9cc9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 82f1641f4cb34d93a53010da960b9cc9, fc610294087b4aaf8b0c4d9dd3d2d474; no voters:
I20260929 20:22:38.684113 19992 raft_consensus.cc:2781] T a70a34f5477a476e89b051729b8aa92d P 82f1641f4cb34d93a53010da960b9cc9 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:22:38.684268 19992 raft_consensus.cc:686] T a70a34f5477a476e89b051729b8aa92d P 82f1641f4cb34d93a53010da960b9cc9 [term 1 LEADER]: Becoming Leader. State: Replica: 82f1641f4cb34d93a53010da960b9cc9, State: Running, Role: LEADER
I20260929 20:22:38.684533 19992 consensus_queue.cc:237] T a70a34f5477a476e89b051729b8aa92d P 82f1641f4cb34d93a53010da960b9cc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba9328e853e140d6b28c75c85f061c69" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 43187 } } peers { permanent_uuid: "fc610294087b4aaf8b0c4d9dd3d2d474" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 43673 } } peers { permanent_uuid: "82f1641f4cb34d93a53010da960b9cc9" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 39893 } }
I20260929 20:22:38.687444 19378 catalog_manager.cc:5495] T a70a34f5477a476e89b051729b8aa92d P 82f1641f4cb34d93a53010da960b9cc9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 82f1641f4cb34d93a53010da960b9cc9 (127.17.53.67). New cstate: current_term: 1 leader_uuid: "82f1641f4cb34d93a53010da960b9cc9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba9328e853e140d6b28c75c85f061c69" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 43187 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fc610294087b4aaf8b0c4d9dd3d2d474" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 43673 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "82f1641f4cb34d93a53010da960b9cc9" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 39893 } health_report { overall_health: HEALTHY } } }
I20260929 20:22:38.716035 19683 raft_consensus.cc:1261] T a70a34f5477a476e89b051729b8aa92d P fc610294087b4aaf8b0c4d9dd3d2d474 [term 1 FOLLOWER]: Refusing update from remote peer 82f1641f4cb34d93a53010da960b9cc9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:22:38.716063 19538 raft_consensus.cc:1261] T a70a34f5477a476e89b051729b8aa92d P ba9328e853e140d6b28c75c85f061c69 [term 1 FOLLOWER]: Refusing update from remote peer 82f1641f4cb34d93a53010da960b9cc9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:22:38.716583 19992 consensus_queue.cc:1035] T a70a34f5477a476e89b051729b8aa92d P 82f1641f4cb34d93a53010da960b9cc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc610294087b4aaf8b0c4d9dd3d2d474" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 43673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:22:38.716846 19997 consensus_queue.cc:1035] T a70a34f5477a476e89b051729b8aa92d P 82f1641f4cb34d93a53010da960b9cc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba9328e853e140d6b28c75c85f061c69" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 43187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260929 20:22:38.737722 19956 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260929 20:22:40.325650 19379 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60632:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 20:22:40.325894 19379 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60632:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 20:22:40.329696 19379 catalog_manager.cc:5782] T 00000000000000000000000000000000 P dd8eef272ec6403e935a5cbc0ef50dce: Sending DeleteTablet for 3 replicas of tablet a70a34f5477a476e89b051729b8aa92d
I20260929 20:22:40.330364 19517 tablet_service.cc:1514] Processing DeleteTablet for tablet a70a34f5477a476e89b051729b8aa92d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:22:40 UTC) from {username='kudu', principal='kudu/127.17.53.126@KRBTEST.COM'} at 127.0.0.1:57486
I20260929 20:22:40.330479 19798 tablet_service.cc:1514] Processing DeleteTablet for tablet a70a34f5477a476e89b051729b8aa92d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:22:40 UTC) from {username='kudu', principal='kudu/127.17.53.126@KRBTEST.COM'} at 127.0.0.1:50368
I20260929 20:22:40.330507 19657 tablet_service.cc:1514] Processing DeleteTablet for tablet a70a34f5477a476e89b051729b8aa92d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:22:40 UTC) from {username='kudu', principal='kudu/127.17.53.126@KRBTEST.COM'} at 127.0.0.1:36066
I20260929 20:22:40.331379 17621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu pid 19451
I20260929 20:22:40.342846 20041 tablet_replica.cc:330] T a70a34f5477a476e89b051729b8aa92d P 82f1641f4cb34d93a53010da960b9cc9: stopping tablet replica
I20260929 20:22:40.343161 20041 raft_consensus.cc:2229] T a70a34f5477a476e89b051729b8aa92d P 82f1641f4cb34d93a53010da960b9cc9 [term 1 LEADER]: Raft consensus shutting down.
I20260929 20:22:40.343662 20041 raft_consensus.cc:2258] T a70a34f5477a476e89b051729b8aa92d P 82f1641f4cb34d93a53010da960b9cc9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:22:40.344393 20040 tablet_replica.cc:330] T a70a34f5477a476e89b051729b8aa92d P ba9328e853e140d6b28c75c85f061c69: stopping tablet replica
I20260929 20:22:40.344600 20040 raft_consensus.cc:2229] T a70a34f5477a476e89b051729b8aa92d P ba9328e853e140d6b28c75c85f061c69 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 20:22:40.344868 20040 raft_consensus.cc:2258] T a70a34f5477a476e89b051729b8aa92d P ba9328e853e140d6b28c75c85f061c69 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:22:40.345628 20041 ts_tablet_manager.cc:1905] T a70a34f5477a476e89b051729b8aa92d P 82f1641f4cb34d93a53010da960b9cc9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:22:40.345927 20042 tablet_replica.cc:330] T a70a34f5477a476e89b051729b8aa92d P fc610294087b4aaf8b0c4d9dd3d2d474: stopping tablet replica
I20260929 20:22:40.346179 20042 raft_consensus.cc:2229] T a70a34f5477a476e89b051729b8aa92d P fc610294087b4aaf8b0c4d9dd3d2d474 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 20:22:40.346441 20042 raft_consensus.cc:2258] T a70a34f5477a476e89b051729b8aa92d P fc610294087b4aaf8b0c4d9dd3d2d474 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:22:40.347504 20042 ts_tablet_manager.cc:1905] T a70a34f5477a476e89b051729b8aa92d P fc610294087b4aaf8b0c4d9dd3d2d474: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:22:40.348155 20040 ts_tablet_manager.cc:1905] T a70a34f5477a476e89b051729b8aa92d P ba9328e853e140d6b28c75c85f061c69: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:22:40.352285 20041 ts_tablet_manager.cc:1918] T a70a34f5477a476e89b051729b8aa92d P 82f1641f4cb34d93a53010da960b9cc9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260929 20:22:40.352465 20041 log.cc:1196] T a70a34f5477a476e89b051729b8aa92d P 82f1641f4cb34d93a53010da960b9cc9: Deleting WAL directory at /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790713339161492-17621-0/minicluster-data/ts-2/wal/wals/a70a34f5477a476e89b051729b8aa92d
I20260929 20:22:40.352950 20041 ts_tablet_manager.cc:1939] T a70a34f5477a476e89b051729b8aa92d P 82f1641f4cb34d93a53010da960b9cc9: Deleting consensus metadata
I20260929 20:22:40.354219 19366 catalog_manager.cc:4841] TS 82f1641f4cb34d93a53010da960b9cc9 (127.17.53.67:39893): tablet a70a34f5477a476e89b051729b8aa92d (table test-table [id=d1131e0947344c01a4b54266196ed965]) successfully deleted
I20260929 20:22:40.354709 20042 ts_tablet_manager.cc:1918] T a70a34f5477a476e89b051729b8aa92d P fc610294087b4aaf8b0c4d9dd3d2d474: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260929 20:22:40.355224 20042 log.cc:1196] T a70a34f5477a476e89b051729b8aa92d P fc610294087b4aaf8b0c4d9dd3d2d474: Deleting WAL directory at /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790713339161492-17621-0/minicluster-data/ts-1/wal/wals/a70a34f5477a476e89b051729b8aa92d
I20260929 20:22:40.355350 20040 ts_tablet_manager.cc:1918] T a70a34f5477a476e89b051729b8aa92d P ba9328e853e140d6b28c75c85f061c69: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260929 20:22:40.355520 20040 log.cc:1196] T a70a34f5477a476e89b051729b8aa92d P ba9328e853e140d6b28c75c85f061c69: Deleting WAL directory at /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790713339161492-17621-0/minicluster-data/ts-0/wal/wals/a70a34f5477a476e89b051729b8aa92d
I20260929 20:22:40.355765 20042 ts_tablet_manager.cc:1939] T a70a34f5477a476e89b051729b8aa92d P fc610294087b4aaf8b0c4d9dd3d2d474: Deleting consensus metadata
I20260929 20:22:40.355983 20040 ts_tablet_manager.cc:1939] T a70a34f5477a476e89b051729b8aa92d P ba9328e853e140d6b28c75c85f061c69: Deleting consensus metadata
I20260929 20:22:40.356784 19364 catalog_manager.cc:4841] TS fc610294087b4aaf8b0c4d9dd3d2d474 (127.17.53.66:43673): tablet a70a34f5477a476e89b051729b8aa92d (table test-table [id=d1131e0947344c01a4b54266196ed965]) successfully deleted
I20260929 20:22:40.356946 19366 catalog_manager.cc:4841] TS ba9328e853e140d6b28c75c85f061c69 (127.17.53.65:43187): tablet a70a34f5477a476e89b051729b8aa92d (table test-table [id=d1131e0947344c01a4b54266196ed965]) successfully deleted
Sep 29 20:22:40 dist-test-slave-zbnc krb5kdc[19334](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713358, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.17.53.65@KRBTEST.COM
I20260929 20:22:40.367195 20044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:40.333093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56542 (local address 127.17.53.65:43187)
0929 20:22:40.347113 (+ 14020us) server_negotiation.cc:206] Beginning negotiation
0929 20:22:40.347122 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 20:22:40.347822 (+ 700us) server_negotiation.cc:407] Connection header received
0929 20:22:40.347888 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:22:40.347893 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:22:40.347984 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:22:40.348081 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:22:40.350014 (+ 1933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:40.351268 (+ 1254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:40.353279 (+ 2011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:40.353968 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:40.354089 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:22:40.361921 (+ 7832us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:22:40.361938 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:22:40.361954 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:22:40.361989 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:22:40.364722 (+ 2733us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:22:40.365334 (+ 612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:22:40.365341 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:22:40.365343 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:22:40.365467 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:22:40.365779 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:22:40.365782 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:22:40.365784 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:22:40.366134 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:22:40.366365 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0929 20:22:40.366599 (+ 234us) server_negotiation.cc:299] Negotiation successful
0929 20:22:40.366777 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13838,"thread_start_us":13749,"threads_started":1}
W20260929 20:22:40.367859 19583 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56542
W20260929 20:22:40.368474 17621 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu pid 19451: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 20:22:40.368731 17621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu with pid 19451
I20260929 20:22:40.394556 17621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu pid 19592
Sep 29 20:22:40 dist-test-slave-zbnc krb5kdc[19334](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713358, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.17.53.66@KRBTEST.COM
I20260929 20:22:40.409653 20045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:40.395348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36168 (local address 127.17.53.66:43673)
0929 20:22:40.395876 (+ 528us) server_negotiation.cc:206] Beginning negotiation
0929 20:22:40.395884 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 20:22:40.395907 (+ 23us) server_negotiation.cc:407] Connection header received
0929 20:22:40.395967 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:22:40.395971 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:22:40.396036 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:22:40.396125 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:22:40.397544 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:40.398276 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:40.399505 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:40.399886 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:40.399979 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:22:40.404211 (+ 4232us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:22:40.404227 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:22:40.404245 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:22:40.404283 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:22:40.407095 (+ 2812us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:22:40.407711 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:22:40.407718 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:22:40.407720 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:22:40.407851 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:22:40.408265 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:22:40.408268 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:22:40.408270 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:22:40.408612 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:22:40.408826 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0929 20:22:40.409086 (+ 260us) server_negotiation.cc:299] Negotiation successful
0929 20:22:40.409280 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":367,"thread_start_us":250,"threads_started":1}
W20260929 20:22:40.410355 19723 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36168
W20260929 20:22:40.411175 17621 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu pid 19592: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 20:22:40.411334 17621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu with pid 19592
I20260929 20:22:40.435216 17621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu pid 19732
Sep 29 20:22:40 dist-test-slave-zbnc krb5kdc[19334](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713358, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.17.53.67@KRBTEST.COM
I20260929 20:22:40.451206 20046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:40.436058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43768 (local address 127.17.53.67:39893)
0929 20:22:40.436554 (+ 496us) server_negotiation.cc:206] Beginning negotiation
0929 20:22:40.436558 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:22:40.436575 (+ 17us) server_negotiation.cc:407] Connection header received
0929 20:22:40.436629 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:22:40.436633 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:22:40.436691 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:22:40.436808 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:22:40.438093 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:40.438832 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:40.440251 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:40.440663 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:40.440769 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:22:40.444925 (+ 4156us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:22:40.444941 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:22:40.444958 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:22:40.444992 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:22:40.448089 (+ 3097us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:22:40.448783 (+ 694us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:22:40.448796 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:22:40.448802 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:22:40.448983 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:22:40.449409 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:22:40.449415 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:22:40.449419 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:22:40.449841 (+ 422us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:22:40.450155 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0929 20:22:40.450485 (+ 330us) server_negotiation.cc:299] Negotiation successful
0929 20:22:40.450715 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":397,"thread_start_us":270,"threads_started":1}
W20260929 20:22:40.452239 19864 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43768
W20260929 20:22:40.452982 17621 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu pid 19732: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 20:22:40.453161 17621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu with pid 19732
I20260929 20:22:40.475797 17621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu pid 19348
I20260929 20:22:40.476960 19420 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:22:40.592195 17621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu with pid 19348
2026-09-29T20:22:40Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (8501 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790713339161492-17621-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 20:22:40 dist-test-slave-zbnc krb5kdc[20050](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 20:22:40 dist-test-slave-zbnc krb5kdc[20050](info): set up 2 sockets
Sep 29 20:22:40 dist-test-slave-zbnc krb5kdc[20050](info): commencing operation
krb5kdc: starting...
W20260929 20:22:41.740492 17621 mini_kdc.cc:121] Time spent starting KDC: real 1.104s user 0.000s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790713339161492-17621-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790713339161492-17621-0/krb5kdc/test-user.keytab.
Sep 29 20:22:41 dist-test-slave-zbnc krb5kdc[20050](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713361, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T20:22:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T20:22:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.17.53.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.53.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.53.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.126.keytab.
Entry for principal kudu/127.17.53.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.126.keytab.
I20260929 20:22:41.934011 17621 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
/tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790713339161492-17621-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790713339161492-17621-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790713339161492-17621-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790713339161492-17621-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.53.126:42127
--webserver_interface=127.17.53.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.126.keytab
--principal=kudu/127.17.53.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.53.84:35383
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.53.126:42127
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790713339161492-17621-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790713339161492-17621-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790713339161492-17621-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:22:42.266136 20063 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:22:42.266438 20063 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:22:42.266667 20063 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:22:42.274066 20063 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:22:42.274185 20063 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:22:42.274303 20063 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:22:42.274411 20063 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:22:42.274515 20063 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:22:42.284555 20063 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.53.84:35383
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790713339161492-17621-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790713339161492-17621-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.53.126:42127
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.126.keytab
--principal=kudu/127.17.53.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.53.126:42127
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790713339161492-17621-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.53.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790713339161492-17621-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:22:42.285980 20063 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:22:42.287387 20063 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:22:42.299926 20068 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:22:42.299890 20071 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:22:42.299840 20069 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:22:42.300724 20063 server_base.cc:1034] running on GCE node
I20260929 20:22:42.301534 20063 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:22:42.303068 20063 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:22:42.304309 20063 hybrid_clock.cc:648] HybridClock initialized: now 1790713362304262 us; error 54 us; skew 500 ppm
I20260929 20:22:42.304672 20063 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:22:42 dist-test-slave-zbnc krb5kdc[20050](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713362, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.53.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:22:42.307826 20063 init.cc:377] Logged in from keytab as kudu/127.17.53.126@KRBTEST.COM (short username kudu)
I20260929 20:22:42.309516 20063 webserver.cc:460] Webserver started at http://127.17.53.126:44435/ using document root <none> and password file <none>
I20260929 20:22:42.310217 20063 fs_manager.cc:359] Metadata directory not provided
I20260929 20:22:42.310350 20063 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:22:42.310676 20063 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:22:42.313766 20063 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790713339161492-17621-0/minicluster-data/master-0/data/instance:
uuid: "4ca539d01d7849929055badf6350153a"
format_stamp: "Formatted at 2026-09-29 20:22:42 on dist-test-slave-zbnc"
server_key: "972644267ca1a3c034fc928c8f88a349"
server_key_iv: "f288b7e6ce7c7a400aac0921d86a0808"
server_key_version: "encryptionkey@0"
I20260929 20:22:42.314504 20063 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790713339161492-17621-0/minicluster-data/master-0/wal/instance:
uuid: "4ca539d01d7849929055badf6350153a"
format_stamp: "Formatted at 2026-09-29 20:22:42 on dist-test-slave-zbnc"
server_key: "972644267ca1a3c034fc928c8f88a349"
server_key_iv: "f288b7e6ce7c7a400aac0921d86a0808"
server_key_version: "encryptionkey@0"
I20260929 20:22:42.319782 20063 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.000s
I20260929 20:22:42.323213 20078 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:22:42.324462 20063 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 20:22:42.324651 20063 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790713339161492-17621-0/minicluster-data/master-0/data,/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790713339161492-17621-0/minicluster-data/master-0/wal
uuid: "4ca539d01d7849929055badf6350153a"
format_stamp: "Formatted at 2026-09-29 20:22:42 on dist-test-slave-zbnc"
server_key: "972644267ca1a3c034fc928c8f88a349"
server_key_iv: "f288b7e6ce7c7a400aac0921d86a0808"
server_key_version: "encryptionkey@0"
I20260929 20:22:42.324849 20063 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790713339161492-17621-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790713339161492-17621-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790713339161492-17621-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:22:42.356756 20063 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:22:42.361208 20063 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:22:42.361515 20063 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:22:42.379954 20063 rpc_server.cc:307] RPC server started. Bound to: 127.17.53.126:42127
I20260929 20:22:42.379978 20130 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.53.126:42127 every 8 connection(s)
I20260929 20:22:42.381397 20063 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790713339161492-17621-0/minicluster-data/master-0/data/info.pb
I20260929 20:22:42.383404 17621 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu as pid 20063
I20260929 20:22:42.383639 17621 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790713339161492-17621-0/minicluster-data/master-0/wal/instance
I20260929 20:22:42.384464 17621 external_mini_cluster.cc:1428] Setting key bd0c6e0c568b89ea1ed6b8a6a5a28963
I20260929 20:22:42.386265 20131 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 29 20:22:42 dist-test-slave-zbnc krb5kdc[20050](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713361, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.53.126@KRBTEST.COM
I20260929 20:22:42.400099 20131 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4ca539d01d7849929055badf6350153a: Bootstrap starting.
I20260929 20:22:42.403332 20131 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4ca539d01d7849929055badf6350153a: Neither blocks nor log segments found. Creating new log.
I20260929 20:22:42.404345 20131 log.cc:824] T 00000000000000000000000000000000 P 4ca539d01d7849929055badf6350153a: Log is configured to *not* fsync() on all Append() calls
I20260929 20:22:42.407209 20134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:42.386585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58984 (local address 127.17.53.126:42127)
0929 20:22:42.387368 (+ 783us) server_negotiation.cc:206] Beginning negotiation
0929 20:22:42.387379 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 20:22:42.387421 (+ 42us) server_negotiation.cc:407] Connection header received
0929 20:22:42.388970 (+ 1549us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:22:42.388991 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:22:42.389367 (+ 376us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:22:42.389692 (+ 325us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:22:42.391496 (+ 1804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:42.392823 (+ 1327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:42.394632 (+ 1809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:42.395333 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:42.395457 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:22:42.400104 (+ 4647us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:22:42.400112 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:22:42.400126 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:22:42.400190 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:22:42.403195 (+ 3005us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:22:42.403948 (+ 753us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:22:42.403954 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:22:42.403960 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:22:42.404127 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:22:42.404529 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:22:42.404535 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:22:42.404538 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:22:42.405029 (+ 491us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:22:42.405358 (+ 329us) server_negotiation.cc:1035] Waiting for connection context
0929 20:22:42.405786 (+ 428us) server_negotiation.cc:299] Negotiation successful
0929 20:22:42.406181 (+ 395us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":592,"thread_start_us":372,"threads_started":1}
I20260929 20:22:42.407434 20131 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4ca539d01d7849929055badf6350153a: No bootstrap required, opened a new log
I20260929 20:22:42.415129 20131 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4ca539d01d7849929055badf6350153a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ca539d01d7849929055badf6350153a" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 42127 } }
I20260929 20:22:42.415591 20131 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4ca539d01d7849929055badf6350153a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:22:42.415701 20131 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4ca539d01d7849929055badf6350153a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ca539d01d7849929055badf6350153a, State: Initialized, Role: FOLLOWER
I20260929 20:22:42.416410 20131 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4ca539d01d7849929055badf6350153a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ca539d01d7849929055badf6350153a" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 42127 } }
I20260929 20:22:42.416684 20131 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4ca539d01d7849929055badf6350153a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 20:22:42.416841 20131 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4ca539d01d7849929055badf6350153a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 20:22:42.417044 20131 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4ca539d01d7849929055badf6350153a [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:22:42.419312 20131 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4ca539d01d7849929055badf6350153a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ca539d01d7849929055badf6350153a" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 42127 } }
I20260929 20:22:42.419888 20131 leader_election.cc:302] T 00000000000000000000000000000000 P 4ca539d01d7849929055badf6350153a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4ca539d01d7849929055badf6350153a; no voters:
I20260929 20:22:42.420465 20131 leader_election.cc:288] T 00000000000000000000000000000000 P 4ca539d01d7849929055badf6350153a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 20:22:42.420629 20136 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4ca539d01d7849929055badf6350153a [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:22:42.421411 20136 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4ca539d01d7849929055badf6350153a [term 1 LEADER]: Becoming Leader. State: Replica: 4ca539d01d7849929055badf6350153a, State: Running, Role: LEADER
I20260929 20:22:42.421967 20136 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4ca539d01d7849929055badf6350153a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ca539d01d7849929055badf6350153a" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 42127 } }
I20260929 20:22:42.422688 20131 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4ca539d01d7849929055badf6350153a [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:22:42.424818 20138 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ca539d01d7849929055badf6350153a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4ca539d01d7849929055badf6350153a. Latest consensus state: current_term: 1 leader_uuid: "4ca539d01d7849929055badf6350153a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ca539d01d7849929055badf6350153a" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 42127 } } }
I20260929 20:22:42.424927 20137 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ca539d01d7849929055badf6350153a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4ca539d01d7849929055badf6350153a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ca539d01d7849929055badf6350153a" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 42127 } } }
I20260929 20:22:42.425205 20138 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ca539d01d7849929055badf6350153a [sys.catalog]: This master's current role is: LEADER
I20260929 20:22:42.425874 20137 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ca539d01d7849929055badf6350153a [sys.catalog]: This master's current role is: LEADER
I20260929 20:22:42.426795 20141 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:22:42.434078 20141 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:22:42.442428 20141 catalog_manager.cc:1347] Generated new cluster ID: 23fb1cf94cd54a85aa467df4a014ead8
I20260929 20:22:42.442548 20141 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:22:42.455484 20141 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 20:22:42.456933 20141 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:22:42.467793 20141 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4ca539d01d7849929055badf6350153a: Generated new TSK 0
I20260929 20:22:42.468621 20141 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.17.53.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.53.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.53.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.65.keytab.
Entry for principal kudu/127.17.53.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.65.keytab.
I20260929 20:22:42.521831 17621 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
/tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790713339161492-17621-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790713339161492-17621-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790713339161492-17621-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790713339161492-17621-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.53.65:0
--local_ip_for_outbound_sockets=127.17.53.65
--webserver_interface=127.17.53.65
--webserver_port=0
--tserver_master_addrs=127.17.53.126:42127
--keytab_file=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.65.keytab
--principal=kudu/127.17.53.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.53.84:35383
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790713339161492-17621-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790713339161492-17621-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790713339161492-17621-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:22:42.846930 20157 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:22:42.847219 20157 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:22:42.847380 20157 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:22:42.854846 20157 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:22:42.854990 20157 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:22:42.855193 20157 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.53.65
I20260929 20:22:42.865036 20157 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.53.84:35383
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790713339161492-17621-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790713339161492-17621-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.65.keytab
--principal=kudu/127.17.53.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.53.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790713339161492-17621-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.17.53.65
--webserver_port=0
--tserver_master_addrs=127.17.53.126:42127
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.53.65
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790713339161492-17621-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:22:42.866376 20157 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:22:42.867784 20157 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:22:42.881145 20165 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:22:42.881227 20162 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:22:42.881203 20163 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:22:42.881681 20157 server_base.cc:1034] running on GCE node
I20260929 20:22:42.882513 20157 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:22:42.883498 20157 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:22:42.884698 20157 hybrid_clock.cc:648] HybridClock initialized: now 1790713362884626 us; error 62 us; skew 500 ppm
I20260929 20:22:42.885032 20157 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:22:42 dist-test-slave-zbnc krb5kdc[20050](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713362, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.53.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:22:42.888257 20157 init.cc:377] Logged in from keytab as kudu/127.17.53.65@KRBTEST.COM (short username kudu)
I20260929 20:22:42.889695 20157 webserver.cc:460] Webserver started at http://127.17.53.65:42603/ using document root <none> and password file <none>
I20260929 20:22:42.890430 20157 fs_manager.cc:359] Metadata directory not provided
I20260929 20:22:42.890578 20157 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:22:42.890966 20157 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:22:42.894001 20157 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790713339161492-17621-0/minicluster-data/ts-0/data/instance:
uuid: "09d43317ec3d4d568491b0ca15a7887e"
format_stamp: "Formatted at 2026-09-29 20:22:42 on dist-test-slave-zbnc"
server_key: "87647b71db4d276516dc73ebfbfcbd60"
server_key_iv: "6c54eae80952ee00c72b67c4f5e150f5"
server_key_version: "encryptionkey@0"
I20260929 20:22:42.894847 20157 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790713339161492-17621-0/minicluster-data/ts-0/wal/instance:
uuid: "09d43317ec3d4d568491b0ca15a7887e"
format_stamp: "Formatted at 2026-09-29 20:22:42 on dist-test-slave-zbnc"
server_key: "87647b71db4d276516dc73ebfbfcbd60"
server_key_iv: "6c54eae80952ee00c72b67c4f5e150f5"
server_key_version: "encryptionkey@0"
I20260929 20:22:42.900293 20157 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260929 20:22:42.903723 20172 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:22:42.905076 20157 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 20:22:42.905267 20157 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790713339161492-17621-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790713339161492-17621-0/minicluster-data/ts-0/wal
uuid: "09d43317ec3d4d568491b0ca15a7887e"
format_stamp: "Formatted at 2026-09-29 20:22:42 on dist-test-slave-zbnc"
server_key: "87647b71db4d276516dc73ebfbfcbd60"
server_key_iv: "6c54eae80952ee00c72b67c4f5e150f5"
server_key_version: "encryptionkey@0"
I20260929 20:22:42.905491 20157 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790713339161492-17621-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790713339161492-17621-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790713339161492-17621-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:22:42.938910 20157 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:22:42.942919 20157 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:22:42.943252 20157 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:22:42.944229 20157 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:22:42.945574 20157 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:22:42.945683 20157 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:22:42.945851 20157 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:22:42.945927 20157 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:22:42.979804 20157 rpc_server.cc:307] RPC server started. Bound to: 127.17.53.65:40301
I20260929 20:22:42.979828 20285 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.53.65:40301 every 8 connection(s)
I20260929 20:22:42.981215 20157 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790713339161492-17621-0/minicluster-data/ts-0/data/info.pb
I20260929 20:22:42.991334 17621 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu as pid 20157
I20260929 20:22:42.991564 17621 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790713339161492-17621-0/minicluster-data/ts-0/wal/instance
I20260929 20:22:42.992285 17621 external_mini_cluster.cc:1428] Setting key ad4e515bf1670d4f3cf659c1d1d6974a
Sep 29 20:22:42 dist-test-slave-zbnc krb5kdc[20050](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713362, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.53.65@KRBTEST.COM for kudu/127.17.53.126@KRBTEST.COM
I20260929 20:22:43.001422 20288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:42.983846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.53.65:54481 (local address 127.17.53.126:42127)
0929 20:22:42.984308 (+ 462us) server_negotiation.cc:206] Beginning negotiation
0929 20:22:42.984312 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:22:42.985284 (+ 972us) server_negotiation.cc:407] Connection header received
0929 20:22:42.987131 (+ 1847us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:22:42.987136 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:22:42.987199 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:22:42.987301 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:22:42.988765 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:42.989480 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:42.991019 (+ 1539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:42.991662 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:42.991809 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:22:42.995650 (+ 3841us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:22:42.995655 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:22:42.995658 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:22:42.995686 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:22:42.998219 (+ 2533us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:22:42.999180 (+ 961us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:22:42.999184 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:22:42.999194 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:22:42.999309 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:22:42.999779 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:22:42.999783 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:22:42.999785 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:22:43.000038 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:22:43.000185 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0929 20:22:43.001002 (+ 817us) server_negotiation.cc:299] Negotiation successful
0929 20:22:43.001164 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":331,"thread_start_us":216,"threads_started":1}
I20260929 20:22:43.003048 20289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:42.984450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:42127 (local address 127.17.53.65:54481)
0929 20:22:42.985110 (+ 660us) negotiation.cc:107] Waiting for socket to connect
0929 20:22:42.985153 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 20:22:42.986811 (+ 1658us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:22:42.987515 (+ 704us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:22:42.987523 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:22:42.988027 (+ 504us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:22:42.988620 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:22:42.988637 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:22:42.989618 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:22:42.989621 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:22:42.990824 (+ 1203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:22:42.990835 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:22:42.991875 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:22:42.991880 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:22:42.992068 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:22:42.992734 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:22:42.992757 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:22:42.995420 (+ 2663us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:22:42.998356 (+ 2936us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:22:42.998367 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:22:42.998386 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:22:42.999041 (+ 655us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:22:42.999428 (+ 387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:22:42.999435 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:22:42.999439 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:22:42.999642 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:22:43.000161 (+ 519us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:22:43.000169 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:22:43.000596 (+ 427us) client_negotiation.cc:769] Sending connection context
0929 20:22:43.000898 (+ 302us) client_negotiation.cc:240] Negotiation successful
0929 20:22:43.001483 (+ 585us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":429,"thread_start_us":267,"threads_started":1}
I20260929 20:22:43.004648 20286 heartbeater.cc:347] Connected to a master server at 127.17.53.126:42127
I20260929 20:22:43.004963 20286 heartbeater.cc:464] Registering TS with master...
I20260929 20:22:43.005692 20286 heartbeater.cc:511] Master 127.17.53.126:42127 requested a full tablet report, sending...
I20260929 20:22:43.008165 20095 ts_manager.cc:194] Registered new tserver with Master: 09d43317ec3d4d568491b0ca15a7887e (127.17.53.65:40301)
I20260929 20:22:43.010454 20095 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.53.65@KRBTEST.COM'} at 127.17.53.65:54481
WARNING: no policy specified for kudu/127.17.53.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.53.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.53.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.66.keytab.
Entry for principal kudu/127.17.53.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.66.keytab.
I20260929 20:22:43.038693 17621 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
/tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790713339161492-17621-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790713339161492-17621-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790713339161492-17621-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790713339161492-17621-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.53.66:0
--local_ip_for_outbound_sockets=127.17.53.66
--webserver_interface=127.17.53.66
--webserver_port=0
--tserver_master_addrs=127.17.53.126:42127
--keytab_file=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.66.keytab
--principal=kudu/127.17.53.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.53.84:35383
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790713339161492-17621-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790713339161492-17621-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790713339161492-17621-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:22:43.369664 20292 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:22:43.369997 20292 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:22:43.370177 20292 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:22:43.377544 20292 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:22:43.377713 20292 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:22:43.377911 20292 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.53.66
I20260929 20:22:43.392777 20292 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.53.84:35383
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790713339161492-17621-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790713339161492-17621-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.66.keytab
--principal=kudu/127.17.53.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.53.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790713339161492-17621-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.17.53.66
--webserver_port=0
--tserver_master_addrs=127.17.53.126:42127
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.53.66
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790713339161492-17621-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:22:43.394430 20292 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:22:43.396111 20292 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:22:43.411123 20297 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:22:43.411154 20298 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:22:43.411255 20300 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:22:43.411902 20292 server_base.cc:1034] running on GCE node
I20260929 20:22:43.412539 20292 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:22:43.413544 20292 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:22:43.414785 20292 hybrid_clock.cc:648] HybridClock initialized: now 1790713363414731 us; error 47 us; skew 500 ppm
I20260929 20:22:43.415145 20292 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:22:43 dist-test-slave-zbnc krb5kdc[20050](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713363, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.53.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:22:43.418318 20292 init.cc:377] Logged in from keytab as kudu/127.17.53.66@KRBTEST.COM (short username kudu)
I20260929 20:22:43.419951 20292 webserver.cc:460] Webserver started at http://127.17.53.66:41963/ using document root <none> and password file <none>
I20260929 20:22:43.420707 20292 fs_manager.cc:359] Metadata directory not provided
I20260929 20:22:43.420851 20292 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:22:43.421169 20292 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:22:43.424420 20292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790713339161492-17621-0/minicluster-data/ts-1/data/instance:
uuid: "839af60e922d424b846b7ae6867458a0"
format_stamp: "Formatted at 2026-09-29 20:22:43 on dist-test-slave-zbnc"
server_key: "e5f9512fcb9abfb1e112b770b4b393e6"
server_key_iv: "3a446911617a273bcb861c960fd64259"
server_key_version: "encryptionkey@0"
I20260929 20:22:43.425215 20292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790713339161492-17621-0/minicluster-data/ts-1/wal/instance:
uuid: "839af60e922d424b846b7ae6867458a0"
format_stamp: "Formatted at 2026-09-29 20:22:43 on dist-test-slave-zbnc"
server_key: "e5f9512fcb9abfb1e112b770b4b393e6"
server_key_iv: "3a446911617a273bcb861c960fd64259"
server_key_version: "encryptionkey@0"
I20260929 20:22:43.430599 20292 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260929 20:22:43.434188 20307 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:22:43.436022 20292 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 20:22:43.436214 20292 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790713339161492-17621-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790713339161492-17621-0/minicluster-data/ts-1/wal
uuid: "839af60e922d424b846b7ae6867458a0"
format_stamp: "Formatted at 2026-09-29 20:22:43 on dist-test-slave-zbnc"
server_key: "e5f9512fcb9abfb1e112b770b4b393e6"
server_key_iv: "3a446911617a273bcb861c960fd64259"
server_key_version: "encryptionkey@0"
I20260929 20:22:43.436430 20292 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790713339161492-17621-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790713339161492-17621-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790713339161492-17621-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:22:43.458297 20292 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:22:43.462222 20292 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:22:43.462535 20292 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:22:43.463537 20292 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:22:43.464913 20292 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:22:43.465022 20292 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:22:43.465175 20292 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:22:43.465250 20292 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:22:43.500718 20292 rpc_server.cc:307] RPC server started. Bound to: 127.17.53.66:41153
I20260929 20:22:43.500770 20420 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.53.66:41153 every 8 connection(s)
I20260929 20:22:43.502133 20292 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790713339161492-17621-0/minicluster-data/ts-1/data/info.pb
I20260929 20:22:43.506933 17621 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu as pid 20292
I20260929 20:22:43.507138 17621 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790713339161492-17621-0/minicluster-data/ts-1/wal/instance
I20260929 20:22:43.507709 17621 external_mini_cluster.cc:1428] Setting key cfd37b05e1b0959bcb389d5a9e99b9cc
Sep 29 20:22:43 dist-test-slave-zbnc krb5kdc[20050](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713363, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.53.66@KRBTEST.COM for kudu/127.17.53.126@KRBTEST.COM
I20260929 20:22:43.525352 20423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:43.504878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.53.66:35701 (local address 127.17.53.126:42127)
0929 20:22:43.505321 (+ 443us) server_negotiation.cc:206] Beginning negotiation
0929 20:22:43.505326 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:22:43.506305 (+ 979us) server_negotiation.cc:407] Connection header received
0929 20:22:43.509022 (+ 2717us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:22:43.509028 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:22:43.509102 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:22:43.509223 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:22:43.511365 (+ 2142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:43.512275 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:43.514900 (+ 2625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:43.515569 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:43.515713 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:22:43.519373 (+ 3660us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:22:43.519379 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:22:43.519385 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:22:43.519422 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:22:43.522130 (+ 2708us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:22:43.522987 (+ 857us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:22:43.522992 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:22:43.523019 (+ 27us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:22:43.523165 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:22:43.523591 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:22:43.523596 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:22:43.523599 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:22:43.523860 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:22:43.524010 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0929 20:22:43.524772 (+ 762us) server_negotiation.cc:299] Negotiation successful
0929 20:22:43.524984 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":323,"thread_start_us":238,"threads_started":1}
WARNING: no policy specified for kudu/127.17.53.67@KRBTEST.COM; defaulting to no policy
I20260929 20:22:43.526396 20424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:43.505433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:42127 (local address 127.17.53.66:35701)
0929 20:22:43.506138 (+ 705us) negotiation.cc:107] Waiting for socket to connect
0929 20:22:43.506178 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0929 20:22:43.508626 (+ 2448us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:22:43.509495 (+ 869us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:22:43.509504 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:22:43.510258 (+ 754us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:22:43.511158 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:22:43.511190 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:22:43.512461 (+ 1271us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:22:43.512470 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:22:43.514708 (+ 2238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:22:43.514727 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:22:43.515844 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:22:43.515852 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:22:43.516187 (+ 335us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:22:43.516964 (+ 777us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:22:43.516987 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:22:43.519144 (+ 2157us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:22:43.522414 (+ 3270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:22:43.522421 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:22:43.522435 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:22:43.522828 (+ 393us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:22:43.523282 (+ 454us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:22:43.523287 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:22:43.523291 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:22:43.523473 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:22:43.523984 (+ 511us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:22:43.523992 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:22:43.524400 (+ 408us) client_negotiation.cc:769] Sending connection context
0929 20:22:43.524667 (+ 267us) client_negotiation.cc:240] Negotiation successful
0929 20:22:43.525181 (+ 514us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":473,"thread_start_us":273,"threads_started":1}
I20260929 20:22:43.527899 20421 heartbeater.cc:347] Connected to a master server at 127.17.53.126:42127
I20260929 20:22:43.528297 20421 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.53.67@KRBTEST.COM" created.
I20260929 20:22:43.529211 20421 heartbeater.cc:511] Master 127.17.53.126:42127 requested a full tablet report, sending...
I20260929 20:22:43.530798 20095 ts_manager.cc:194] Registered new tserver with Master: 839af60e922d424b846b7ae6867458a0 (127.17.53.66:41153)
I20260929 20:22:43.531631 20095 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.53.66@KRBTEST.COM'} at 127.17.53.66:35701
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.53.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.67.keytab.
Entry for principal kudu/127.17.53.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.67.keytab.
I20260929 20:22:43.554271 17621 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
/tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790713339161492-17621-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790713339161492-17621-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790713339161492-17621-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790713339161492-17621-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.53.67:0
--local_ip_for_outbound_sockets=127.17.53.67
--webserver_interface=127.17.53.67
--webserver_port=0
--tserver_master_addrs=127.17.53.126:42127
--keytab_file=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.67.keytab
--principal=kudu/127.17.53.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.53.84:35383
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790713339161492-17621-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790713339161492-17621-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790713339161492-17621-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:22:43.879624 20427 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:22:43.879974 20427 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:22:43.880156 20427 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:22:43.887370 20427 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:22:43.887542 20427 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:22:43.887735 20427 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.53.67
I20260929 20:22:43.897794 20427 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.53.84:35383
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790713339161492-17621-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790713339161492-17621-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.67.keytab
--principal=kudu/127.17.53.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.53.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790713339161492-17621-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.17.53.67
--webserver_port=0
--tserver_master_addrs=127.17.53.126:42127
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.53.67
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790713339161492-17621-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:22:43.899209 20427 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:22:43.900532 20427 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:22:43.914009 20432 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:22:43.913975 20435 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:22:43.913966 20433 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:22:43.915304 20427 server_base.cc:1034] running on GCE node
I20260929 20:22:43.915951 20427 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:22:43.916867 20427 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:22:43.918092 20427 hybrid_clock.cc:648] HybridClock initialized: now 1790713363918039 us; error 51 us; skew 500 ppm
I20260929 20:22:43.918416 20427 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:22:43 dist-test-slave-zbnc krb5kdc[20050](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713363, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.53.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:22:43.921556 20427 init.cc:377] Logged in from keytab as kudu/127.17.53.67@KRBTEST.COM (short username kudu)
I20260929 20:22:43.923203 20427 webserver.cc:460] Webserver started at http://127.17.53.67:37123/ using document root <none> and password file <none>
I20260929 20:22:43.923924 20427 fs_manager.cc:359] Metadata directory not provided
I20260929 20:22:43.924080 20427 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:22:43.924383 20427 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:22:43.927340 20427 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790713339161492-17621-0/minicluster-data/ts-2/data/instance:
uuid: "386c25deae8a4c0786ff4ceb8c9e31b6"
format_stamp: "Formatted at 2026-09-29 20:22:43 on dist-test-slave-zbnc"
server_key: "8bfc2e5e5470df01e80c0ee9fff53a00"
server_key_iv: "699639a7d8c41b0b6a91bdb069d79821"
server_key_version: "encryptionkey@0"
I20260929 20:22:43.928247 20427 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790713339161492-17621-0/minicluster-data/ts-2/wal/instance:
uuid: "386c25deae8a4c0786ff4ceb8c9e31b6"
format_stamp: "Formatted at 2026-09-29 20:22:43 on dist-test-slave-zbnc"
server_key: "8bfc2e5e5470df01e80c0ee9fff53a00"
server_key_iv: "699639a7d8c41b0b6a91bdb069d79821"
server_key_version: "encryptionkey@0"
I20260929 20:22:43.933707 20427 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260929 20:22:43.937559 20442 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:22:43.938956 20427 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260929 20:22:43.939112 20427 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790713339161492-17621-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790713339161492-17621-0/minicluster-data/ts-2/wal
uuid: "386c25deae8a4c0786ff4ceb8c9e31b6"
format_stamp: "Formatted at 2026-09-29 20:22:43 on dist-test-slave-zbnc"
server_key: "8bfc2e5e5470df01e80c0ee9fff53a00"
server_key_iv: "699639a7d8c41b0b6a91bdb069d79821"
server_key_version: "encryptionkey@0"
I20260929 20:22:43.939323 20427 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790713339161492-17621-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790713339161492-17621-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790713339161492-17621-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:22:43.961728 20427 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:22:43.965740 20427 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:22:43.966037 20427 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:22:43.967154 20427 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:22:43.968536 20427 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:22:43.968612 20427 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:22:43.968760 20427 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:22:43.968822 20427 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:22:44.003082 20427 rpc_server.cc:307] RPC server started. Bound to: 127.17.53.67:43339
I20260929 20:22:44.003124 20554 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.53.67:43339 every 8 connection(s)
I20260929 20:22:44.004482 20427 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790713339161492-17621-0/minicluster-data/ts-2/data/info.pb
I20260929 20:22:44.012857 17621 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu as pid 20427
I20260929 20:22:44.013047 17621 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790713339161492-17621-0/minicluster-data/ts-2/wal/instance
I20260929 20:22:44.013581 17621 external_mini_cluster.cc:1428] Setting key a1d604747e5af52bc22624c3d5df102a
I20260929 20:22:44.013769 20286 heartbeater.cc:503] Master 127.17.53.126:42127 was elected leader, sending a full tablet report...
Sep 29 20:22:44 dist-test-slave-zbnc krb5kdc[20050](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713363, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.53.67@KRBTEST.COM for kudu/127.17.53.126@KRBTEST.COM
I20260929 20:22:44.024942 20423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:44.007098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.53.67:51469 (local address 127.17.53.126:42127)
0929 20:22:44.007304 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0929 20:22:44.007308 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:22:44.008593 (+ 1285us) server_negotiation.cc:407] Connection header received
0929 20:22:44.010549 (+ 1956us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:22:44.010554 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:22:44.010665 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:22:44.010777 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:22:44.012429 (+ 1652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:44.013302 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:44.015510 (+ 2208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:44.015972 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:44.016071 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:22:44.019637 (+ 3566us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:22:44.019642 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:22:44.019646 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:22:44.019675 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:22:44.022063 (+ 2388us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:22:44.022730 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:22:44.022735 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:22:44.022745 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:22:44.022911 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:22:44.023413 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:22:44.023416 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:22:44.023418 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:22:44.023676 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:22:44.023810 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0929 20:22:44.024528 (+ 718us) server_negotiation.cc:299] Negotiation successful
0929 20:22:44.024700 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260929 20:22:44.025986 20557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:44.007674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:42127 (local address 127.17.53.67:51469)
0929 20:22:44.008426 (+ 752us) negotiation.cc:107] Waiting for socket to connect
0929 20:22:44.008468 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 20:22:44.010290 (+ 1822us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:22:44.011034 (+ 744us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:22:44.011042 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:22:44.011688 (+ 646us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:22:44.012265 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:22:44.012282 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:22:44.013443 (+ 1161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:22:44.013455 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:22:44.015380 (+ 1925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:22:44.015398 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:22:44.016201 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:22:44.016208 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:22:44.016504 (+ 296us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:22:44.017232 (+ 728us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:22:44.017256 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:22:44.019434 (+ 2178us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:22:44.022211 (+ 2777us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:22:44.022218 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:22:44.022231 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:22:44.022622 (+ 391us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:22:44.023032 (+ 410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:22:44.023036 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:22:44.023039 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:22:44.023307 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:22:44.023807 (+ 500us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:22:44.023813 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:22:44.024117 (+ 304us) client_negotiation.cc:769] Sending connection context
0929 20:22:44.024432 (+ 315us) client_negotiation.cc:240] Negotiation successful
0929 20:22:44.024827 (+ 395us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":518,"thread_start_us":278,"threads_started":1}
I20260929 20:22:44.027527 20555 heartbeater.cc:347] Connected to a master server at 127.17.53.126:42127
I20260929 20:22:44.027889 20555 heartbeater.cc:464] Registering TS with master...
I20260929 20:22:44.028878 20555 heartbeater.cc:511] Master 127.17.53.126:42127 requested a full tablet report, sending...
I20260929 20:22:44.030303 20095 ts_manager.cc:194] Registered new tserver with Master: 386c25deae8a4c0786ff4ceb8c9e31b6 (127.17.53.67:43339)
I20260929 20:22:44.031200 20095 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.53.67@KRBTEST.COM'} at 127.17.53.67:51469
I20260929 20:22:44.041473 17621 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260929 20:22:44.072137 20423 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:22:44.069753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59000 (local address 127.17.53.126:42127)
0929 20:22:44.069922 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0929 20:22:44.069927 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:22:44.070383 (+ 456us) server_negotiation.cc:407] Connection header received
0929 20:22:44.070535 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:22:44.070539 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:22:44.070609 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:22:44.070704 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:22:44.071940 (+ 1236us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59000: BlockingRecv error: recv got EOF from 127.0.0.1:59000 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 20:22:44.075373 17621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu pid 20157
W20260929 20:22:44.078118 17621 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu pid 20157: Not authorized: Client connection negotiation failed: client connection to 127.17.53.65:40301: server requires authentication, but client does not have Kerberos credentials available
I20260929 20:22:44.078310 17621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu with pid 20157
I20260929 20:22:44.094075 17621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu pid 20292
W20260929 20:22:44.096741 17621 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu pid 20292: Not authorized: Client connection negotiation failed: client connection to 127.17.53.66:41153: server requires authentication, but client does not have Kerberos credentials available
I20260929 20:22:44.096925 17621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu with pid 20292
I20260929 20:22:44.112717 17621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu pid 20427
W20260929 20:22:44.115496 17621 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu pid 20427: Not authorized: Client connection negotiation failed: client connection to 127.17.53.67:43339: server requires authentication, but client does not have Kerberos credentials available
I20260929 20:22:44.115658 17621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu with pid 20427
I20260929 20:22:44.131525 17621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu pid 20063
I20260929 20:22:44.132779 20126 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:22:44.234476 17621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu with pid 20063
2026-09-29T20:22:44Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (3639 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790713339161492-17621-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 20:22:44 dist-test-slave-zbnc krb5kdc[20573](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 20:22:44 dist-test-slave-zbnc krb5kdc[20573](info): set up 2 sockets
Sep 29 20:22:44 dist-test-slave-zbnc krb5kdc[20573](info): commencing operation
krb5kdc: starting...
W20260929 20:22:45.356590 17621 mini_kdc.cc:121] Time spent starting KDC: real 1.080s user 0.000s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790713339161492-17621-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790713339161492-17621-0/krb5kdc/test-user.keytab.
Sep 29 20:22:45 dist-test-slave-zbnc krb5kdc[20573](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713365, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T20:22:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T20:22:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.17.53.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.53.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.53.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.126.keytab.
Entry for principal kudu/127.17.53.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.126.keytab.
I20260929 20:22:45.554198 17621 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
/tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790713339161492-17621-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790713339161492-17621-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790713339161492-17621-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790713339161492-17621-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.53.126:33803
--webserver_interface=127.17.53.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.126.keytab
--principal=kudu/127.17.53.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.53.84:36281
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.53.126:33803
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790713339161492-17621-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790713339161492-17621-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790713339161492-17621-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:22:45.883342 20586 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:22:45.883695 20586 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:22:45.883875 20586 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:22:45.891382 20586 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:22:45.891525 20586 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:22:45.891633 20586 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:22:45.891741 20586 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:22:45.891827 20586 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:22:45.901942 20586 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.53.84:36281
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790713339161492-17621-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790713339161492-17621-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.53.126:33803
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.126.keytab
--principal=kudu/127.17.53.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.53.126:33803
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790713339161492-17621-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.53.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790713339161492-17621-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:22:45.903389 20586 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:22:45.904723 20586 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:22:45.916688 20594 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:22:45.916775 20592 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:22:45.916666 20591 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:22:45.917945 20586 server_base.cc:1034] running on GCE node
I20260929 20:22:45.918720 20586 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:22:45.920154 20586 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:22:45.921360 20586 hybrid_clock.cc:648] HybridClock initialized: now 1790713365921320 us; error 29 us; skew 500 ppm
I20260929 20:22:45.921727 20586 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:22:45 dist-test-slave-zbnc krb5kdc[20573](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713365, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.53.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:22:45.930693 20586 init.cc:377] Logged in from keytab as kudu/127.17.53.126@KRBTEST.COM (short username kudu)
I20260929 20:22:45.932250 20586 webserver.cc:460] Webserver started at http://127.17.53.126:37761/ using document root <none> and password file <none>
I20260929 20:22:45.932967 20586 fs_manager.cc:359] Metadata directory not provided
I20260929 20:22:45.933103 20586 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:22:45.933442 20586 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:22:45.936527 20586 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790713339161492-17621-0/minicluster-data/master-0/data/instance:
uuid: "e00e8663cf9247f9b822f2acdb320a86"
format_stamp: "Formatted at 2026-09-29 20:22:45 on dist-test-slave-zbnc"
server_key: "7bcf201a1c92a31c9d103910dda0cffa"
server_key_iv: "69f7652383d1ac5a8d4b51eaea33fc0c"
server_key_version: "encryptionkey@0"
I20260929 20:22:45.937386 20586 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790713339161492-17621-0/minicluster-data/master-0/wal/instance:
uuid: "e00e8663cf9247f9b822f2acdb320a86"
format_stamp: "Formatted at 2026-09-29 20:22:45 on dist-test-slave-zbnc"
server_key: "7bcf201a1c92a31c9d103910dda0cffa"
server_key_iv: "69f7652383d1ac5a8d4b51eaea33fc0c"
server_key_version: "encryptionkey@0"
I20260929 20:22:45.942996 20586 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260929 20:22:45.946970 20602 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:22:45.948302 20586 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260929 20:22:45.948480 20586 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790713339161492-17621-0/minicluster-data/master-0/data,/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790713339161492-17621-0/minicluster-data/master-0/wal
uuid: "e00e8663cf9247f9b822f2acdb320a86"
format_stamp: "Formatted at 2026-09-29 20:22:45 on dist-test-slave-zbnc"
server_key: "7bcf201a1c92a31c9d103910dda0cffa"
server_key_iv: "69f7652383d1ac5a8d4b51eaea33fc0c"
server_key_version: "encryptionkey@0"
I20260929 20:22:45.948662 20586 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790713339161492-17621-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790713339161492-17621-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790713339161492-17621-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:22:45.995397 20586 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:22:45.999931 20586 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:22:46.000221 20586 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:22:46.019043 20586 rpc_server.cc:307] RPC server started. Bound to: 127.17.53.126:33803
I20260929 20:22:46.019063 20653 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.53.126:33803 every 8 connection(s)
I20260929 20:22:46.020422 20586 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790713339161492-17621-0/minicluster-data/master-0/data/info.pb
I20260929 20:22:46.023559 17621 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu as pid 20586
I20260929 20:22:46.023787 17621 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790713339161492-17621-0/minicluster-data/master-0/wal/instance
I20260929 20:22:46.024600 17621 external_mini_cluster.cc:1428] Setting key 51e50a3036b88936b73a133af78ae5d0
I20260929 20:22:46.024897 20654 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:22:46.039762 20654 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e00e8663cf9247f9b822f2acdb320a86: Bootstrap starting.
Sep 29 20:22:46 dist-test-slave-zbnc krb5kdc[20573](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713365, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.53.126@KRBTEST.COM
I20260929 20:22:46.043118 20654 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e00e8663cf9247f9b822f2acdb320a86: Neither blocks nor log segments found. Creating new log.
I20260929 20:22:46.044171 20654 log.cc:824] T 00000000000000000000000000000000 P e00e8663cf9247f9b822f2acdb320a86: Log is configured to *not* fsync() on all Append() calls
I20260929 20:22:46.047017 20654 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e00e8663cf9247f9b822f2acdb320a86: No bootstrap required, opened a new log
I20260929 20:22:46.048399 20657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:46.026506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47514 (local address 127.17.53.126:33803)
0929 20:22:46.027338 (+ 832us) server_negotiation.cc:206] Beginning negotiation
0929 20:22:46.027349 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 20:22:46.027402 (+ 53us) server_negotiation.cc:407] Connection header received
0929 20:22:46.029710 (+ 2308us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:22:46.029733 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:22:46.030139 (+ 406us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:22:46.030487 (+ 348us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:22:46.032440 (+ 1953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:46.033726 (+ 1286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:46.035537 (+ 1811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:46.036265 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:46.036405 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:22:46.040962 (+ 4557us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:22:46.040970 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:22:46.040985 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:22:46.041027 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:22:46.044301 (+ 3274us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:22:46.045006 (+ 705us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:22:46.045013 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:22:46.045019 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:22:46.045190 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:22:46.045603 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:22:46.045607 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:22:46.045609 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:22:46.046107 (+ 498us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:22:46.046412 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0929 20:22:46.046878 (+ 466us) server_negotiation.cc:299] Negotiation successful
0929 20:22:46.047433 (+ 555us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":630,"thread_start_us":417,"threads_started":1}
I20260929 20:22:46.054981 17621 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 20:22:46.059561 20654 raft_consensus.cc:348] T 00000000000000000000000000000000 P e00e8663cf9247f9b822f2acdb320a86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e00e8663cf9247f9b822f2acdb320a86" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 33803 } }
I20260929 20:22:46.060094 20654 raft_consensus.cc:374] T 00000000000000000000000000000000 P e00e8663cf9247f9b822f2acdb320a86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:22:46.060352 20654 raft_consensus.cc:729] T 00000000000000000000000000000000 P e00e8663cf9247f9b822f2acdb320a86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e00e8663cf9247f9b822f2acdb320a86, State: Initialized, Role: FOLLOWER
I20260929 20:22:46.061064 20654 consensus_queue.cc:260] T 00000000000000000000000000000000 P e00e8663cf9247f9b822f2acdb320a86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e00e8663cf9247f9b822f2acdb320a86" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 33803 } }
I20260929 20:22:46.061388 20654 raft_consensus.cc:388] T 00000000000000000000000000000000 P e00e8663cf9247f9b822f2acdb320a86 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 20:22:46.061601 20654 raft_consensus.cc:482] T 00000000000000000000000000000000 P e00e8663cf9247f9b822f2acdb320a86 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 20:22:46.061872 20654 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e00e8663cf9247f9b822f2acdb320a86 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:22:46.065999 20654 raft_consensus.cc:504] T 00000000000000000000000000000000 P e00e8663cf9247f9b822f2acdb320a86 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e00e8663cf9247f9b822f2acdb320a86" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 33803 } }
I20260929 20:22:46.066675 20654 leader_election.cc:302] T 00000000000000000000000000000000 P e00e8663cf9247f9b822f2acdb320a86 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e00e8663cf9247f9b822f2acdb320a86; no voters:
I20260929 20:22:46.067538 20654 leader_election.cc:288] T 00000000000000000000000000000000 P e00e8663cf9247f9b822f2acdb320a86 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 20:22:46.067767 20660 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e00e8663cf9247f9b822f2acdb320a86 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:22:46.069965 20654 sys_catalog.cc:564] T 00000000000000000000000000000000 P e00e8663cf9247f9b822f2acdb320a86 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:22:46.073441 20660 raft_consensus.cc:686] T 00000000000000000000000000000000 P e00e8663cf9247f9b822f2acdb320a86 [term 1 LEADER]: Becoming Leader. State: Replica: e00e8663cf9247f9b822f2acdb320a86, State: Running, Role: LEADER
I20260929 20:22:46.074160 20660 consensus_queue.cc:237] T 00000000000000000000000000000000 P e00e8663cf9247f9b822f2acdb320a86 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e00e8663cf9247f9b822f2acdb320a86" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 33803 } }
I20260929 20:22:46.083178 20661 sys_catalog.cc:455] T 00000000000000000000000000000000 P e00e8663cf9247f9b822f2acdb320a86 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e00e8663cf9247f9b822f2acdb320a86" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e00e8663cf9247f9b822f2acdb320a86" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 33803 } } }
I20260929 20:22:46.083585 20661 sys_catalog.cc:458] T 00000000000000000000000000000000 P e00e8663cf9247f9b822f2acdb320a86 [sys.catalog]: This master's current role is: LEADER
I20260929 20:22:46.084262 20662 sys_catalog.cc:455] T 00000000000000000000000000000000 P e00e8663cf9247f9b822f2acdb320a86 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e00e8663cf9247f9b822f2acdb320a86. Latest consensus state: current_term: 1 leader_uuid: "e00e8663cf9247f9b822f2acdb320a86" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e00e8663cf9247f9b822f2acdb320a86" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 33803 } } }
I20260929 20:22:46.085012 20662 sys_catalog.cc:458] T 00000000000000000000000000000000 P e00e8663cf9247f9b822f2acdb320a86 [sys.catalog]: This master's current role is: LEADER
I20260929 20:22:46.092582 20674 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:22:46.099197 20674 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:22:46.108606 20674 catalog_manager.cc:1347] Generated new cluster ID: f2e803ff890840bb97757dd0b5a59937
I20260929 20:22:46.108747 20674 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:22:46.114384 20657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:46.105157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47522 (local address 127.17.53.126:33803)
0929 20:22:46.105319 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0929 20:22:46.105323 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:22:46.106023 (+ 700us) server_negotiation.cc:407] Connection header received
0929 20:22:46.106231 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:22:46.106240 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:22:46.106352 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:22:46.106482 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:22:46.107629 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:46.108681 (+ 1052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:46.110540 (+ 1859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:46.111246 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:46.111354 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:22:46.112802 (+ 1448us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:22:46.112809 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:22:46.112813 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:22:46.113580 (+ 767us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:22:46.113635 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:22:46.113668 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:22:46.113768 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0929 20:22:46.113946 (+ 178us) server_negotiation.cc:299] Negotiation successful
0929 20:22:46.114032 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260929 20:22:46.121814 20674 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 20:22:46.123708 20674 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:22:46.133080 20674 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e00e8663cf9247f9b822f2acdb320a86: Generated new TSK 0
I20260929 20:22:46.134168 20674 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 20:22:46.151463 17621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu pid 20586
I20260929 20:22:46.153790 20649 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:22:46.243175 17621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu with pid 20586
2026-09-29T20:22:46Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (2002 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790713339161492-17621-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 20:22:46 dist-test-slave-zbnc krb5kdc[20689](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 20:22:46 dist-test-slave-zbnc krb5kdc[20689](info): set up 2 sockets
Sep 29 20:22:46 dist-test-slave-zbnc krb5kdc[20689](info): commencing operation
krb5kdc: starting...
W20260929 20:22:47.328751 17621 mini_kdc.cc:121] Time spent starting KDC: real 1.050s user 0.001s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790713339161492-17621-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790713339161492-17621-0/krb5kdc/test-user.keytab.
Sep 29 20:22:47 dist-test-slave-zbnc krb5kdc[20689](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713367, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T20:22:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T20:22:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.17.53.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.53.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.53.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.126.keytab.
Entry for principal kudu/127.17.53.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.126.keytab.
I20260929 20:22:47.522534 17621 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
/tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790713339161492-17621-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790713339161492-17621-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790713339161492-17621-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790713339161492-17621-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.53.126:38699
--webserver_interface=127.17.53.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.126.keytab
--principal=kudu/127.17.53.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.53.84:36353
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.53.126:38699
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790713339161492-17621-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790713339161492-17621-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790713339161492-17621-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:22:47.840147 20702 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:22:47.840425 20702 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:22:47.840512 20702 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:22:47.847591 20702 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:22:47.847705 20702 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:22:47.847752 20702 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:22:47.847810 20702 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:22:47.847853 20702 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:22:47.857491 20702 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.53.84:36353
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790713339161492-17621-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790713339161492-17621-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.53.126:38699
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.126.keytab
--principal=kudu/127.17.53.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.53.126:38699
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790713339161492-17621-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.53.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790713339161492-17621-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:22:47.858804 20702 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:22:47.860069 20702 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:22:47.871958 20707 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:22:47.871905 20708 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:22:47.872126 20710 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:22:47.873121 20702 server_base.cc:1034] running on GCE node
I20260929 20:22:47.873896 20702 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:22:47.875640 20702 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:22:47.876885 20702 hybrid_clock.cc:648] HybridClock initialized: now 1790713367876834 us; error 57 us; skew 500 ppm
I20260929 20:22:47.877246 20702 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:22:47 dist-test-slave-zbnc krb5kdc[20689](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713367, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.53.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:22:47.880673 20702 init.cc:377] Logged in from keytab as kudu/127.17.53.126@KRBTEST.COM (short username kudu)
I20260929 20:22:47.882149 20702 webserver.cc:460] Webserver started at http://127.17.53.126:39313/ using document root <none> and password file <none>
I20260929 20:22:47.882844 20702 fs_manager.cc:359] Metadata directory not provided
I20260929 20:22:47.883050 20702 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:22:47.883385 20702 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:22:47.886451 20702 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790713339161492-17621-0/minicluster-data/master-0/data/instance:
uuid: "c818db7dfdaf4f5a846abd3f17c25173"
format_stamp: "Formatted at 2026-09-29 20:22:47 on dist-test-slave-zbnc"
server_key: "726b9e3c4b25b8bd2192f91ec2ecda00"
server_key_iv: "1ce8af6de5fbf7a02ddc3b447babcd6f"
server_key_version: "encryptionkey@0"
I20260929 20:22:47.887270 20702 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790713339161492-17621-0/minicluster-data/master-0/wal/instance:
uuid: "c818db7dfdaf4f5a846abd3f17c25173"
format_stamp: "Formatted at 2026-09-29 20:22:47 on dist-test-slave-zbnc"
server_key: "726b9e3c4b25b8bd2192f91ec2ecda00"
server_key_iv: "1ce8af6de5fbf7a02ddc3b447babcd6f"
server_key_version: "encryptionkey@0"
I20260929 20:22:47.892658 20702 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260929 20:22:47.896354 20717 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:22:47.897941 20702 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 20:22:47.898159 20702 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790713339161492-17621-0/minicluster-data/master-0/data,/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790713339161492-17621-0/minicluster-data/master-0/wal
uuid: "c818db7dfdaf4f5a846abd3f17c25173"
format_stamp: "Formatted at 2026-09-29 20:22:47 on dist-test-slave-zbnc"
server_key: "726b9e3c4b25b8bd2192f91ec2ecda00"
server_key_iv: "1ce8af6de5fbf7a02ddc3b447babcd6f"
server_key_version: "encryptionkey@0"
I20260929 20:22:47.898381 20702 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790713339161492-17621-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790713339161492-17621-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790713339161492-17621-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:22:47.925254 20702 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:22:47.929708 20702 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:22:47.929979 20702 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:22:47.948127 20702 rpc_server.cc:307] RPC server started. Bound to: 127.17.53.126:38699
I20260929 20:22:47.948160 20769 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.53.126:38699 every 8 connection(s)
I20260929 20:22:47.949496 20702 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790713339161492-17621-0/minicluster-data/master-0/data/info.pb
I20260929 20:22:47.950573 17621 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu as pid 20702
I20260929 20:22:47.950814 17621 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790713339161492-17621-0/minicluster-data/master-0/wal/instance
I20260929 20:22:47.951474 17621 external_mini_cluster.cc:1428] Setting key 5841b416610f92970bb8d334e8c6f02a
I20260929 20:22:47.953858 20770 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:22:47.964416 20770 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c818db7dfdaf4f5a846abd3f17c25173: Bootstrap starting.
Sep 29 20:22:47 dist-test-slave-zbnc krb5kdc[20689](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713367, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.53.126@KRBTEST.COM
I20260929 20:22:47.967934 20770 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c818db7dfdaf4f5a846abd3f17c25173: Neither blocks nor log segments found. Creating new log.
I20260929 20:22:47.968966 20770 log.cc:824] T 00000000000000000000000000000000 P c818db7dfdaf4f5a846abd3f17c25173: Log is configured to *not* fsync() on all Append() calls
I20260929 20:22:47.972272 20770 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c818db7dfdaf4f5a846abd3f17c25173: No bootstrap required, opened a new log
I20260929 20:22:47.973361 20773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:47.953502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41808 (local address 127.17.53.126:38699)
0929 20:22:47.954290 (+ 788us) server_negotiation.cc:206] Beginning negotiation
0929 20:22:47.954302 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 20:22:47.954340 (+ 38us) server_negotiation.cc:407] Connection header received
0929 20:22:47.956042 (+ 1702us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:22:47.956062 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:22:47.956439 (+ 377us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:22:47.956863 (+ 424us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:22:47.958262 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:47.959733 (+ 1471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:47.960964 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:47.961650 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:47.961875 (+ 225us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:22:47.966080 (+ 4205us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:22:47.966088 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:22:47.966106 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:22:47.966144 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:22:47.969301 (+ 3157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:22:47.970052 (+ 751us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:22:47.970067 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:22:47.970077 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:22:47.970273 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:22:47.970698 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:22:47.970701 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:22:47.970704 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:22:47.971259 (+ 555us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:22:47.971503 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0929 20:22:47.971934 (+ 431us) server_negotiation.cc:299] Negotiation successful
0929 20:22:47.972272 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":586,"thread_start_us":342,"threads_started":1}
I20260929 20:22:47.980211 20770 raft_consensus.cc:348] T 00000000000000000000000000000000 P c818db7dfdaf4f5a846abd3f17c25173 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c818db7dfdaf4f5a846abd3f17c25173" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 38699 } }
I20260929 20:22:47.980581 20770 raft_consensus.cc:374] T 00000000000000000000000000000000 P c818db7dfdaf4f5a846abd3f17c25173 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:22:47.980711 20770 raft_consensus.cc:729] T 00000000000000000000000000000000 P c818db7dfdaf4f5a846abd3f17c25173 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c818db7dfdaf4f5a846abd3f17c25173, State: Initialized, Role: FOLLOWER
I20260929 20:22:47.981335 20770 consensus_queue.cc:260] T 00000000000000000000000000000000 P c818db7dfdaf4f5a846abd3f17c25173 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c818db7dfdaf4f5a846abd3f17c25173" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 38699 } }
I20260929 20:22:47.981529 20770 raft_consensus.cc:388] T 00000000000000000000000000000000 P c818db7dfdaf4f5a846abd3f17c25173 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 20:22:47.981637 20770 raft_consensus.cc:482] T 00000000000000000000000000000000 P c818db7dfdaf4f5a846abd3f17c25173 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 20:22:47.981768 20770 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c818db7dfdaf4f5a846abd3f17c25173 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:22:47.983992 20770 raft_consensus.cc:504] T 00000000000000000000000000000000 P c818db7dfdaf4f5a846abd3f17c25173 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c818db7dfdaf4f5a846abd3f17c25173" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 38699 } }
I20260929 20:22:47.984480 20770 leader_election.cc:302] T 00000000000000000000000000000000 P c818db7dfdaf4f5a846abd3f17c25173 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c818db7dfdaf4f5a846abd3f17c25173; no voters:
I20260929 20:22:47.985088 20770 leader_election.cc:288] T 00000000000000000000000000000000 P c818db7dfdaf4f5a846abd3f17c25173 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 20:22:47.985244 20775 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c818db7dfdaf4f5a846abd3f17c25173 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:22:47.985926 20775 raft_consensus.cc:686] T 00000000000000000000000000000000 P c818db7dfdaf4f5a846abd3f17c25173 [term 1 LEADER]: Becoming Leader. State: Replica: c818db7dfdaf4f5a846abd3f17c25173, State: Running, Role: LEADER
I20260929 20:22:47.986442 20775 consensus_queue.cc:237] T 00000000000000000000000000000000 P c818db7dfdaf4f5a846abd3f17c25173 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c818db7dfdaf4f5a846abd3f17c25173" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 38699 } }
I20260929 20:22:47.987172 20770 sys_catalog.cc:564] T 00000000000000000000000000000000 P c818db7dfdaf4f5a846abd3f17c25173 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:22:47.989178 20776 sys_catalog.cc:455] T 00000000000000000000000000000000 P c818db7dfdaf4f5a846abd3f17c25173 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c818db7dfdaf4f5a846abd3f17c25173" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c818db7dfdaf4f5a846abd3f17c25173" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 38699 } } }
I20260929 20:22:47.989495 20776 sys_catalog.cc:458] T 00000000000000000000000000000000 P c818db7dfdaf4f5a846abd3f17c25173 [sys.catalog]: This master's current role is: LEADER
I20260929 20:22:47.989948 20777 sys_catalog.cc:455] T 00000000000000000000000000000000 P c818db7dfdaf4f5a846abd3f17c25173 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c818db7dfdaf4f5a846abd3f17c25173. Latest consensus state: current_term: 1 leader_uuid: "c818db7dfdaf4f5a846abd3f17c25173" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c818db7dfdaf4f5a846abd3f17c25173" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 38699 } } }
I20260929 20:22:47.990442 20777 sys_catalog.cc:458] T 00000000000000000000000000000000 P c818db7dfdaf4f5a846abd3f17c25173 [sys.catalog]: This master's current role is: LEADER
I20260929 20:22:47.992067 20780 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:22:47.997579 20780 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:22:48.007419 20780 catalog_manager.cc:1347] Generated new cluster ID: d7f6d155761a43d79b224496af615a25
I20260929 20:22:48.007557 20780 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:22:48.037734 20780 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 20:22:48.039677 20780 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:22:48.049826 20780 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c818db7dfdaf4f5a846abd3f17c25173: Generated new TSK 0
I20260929 20:22:48.051100 20780 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.17.53.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.53.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.53.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.65.keytab.
Entry for principal kudu/127.17.53.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.65.keytab.
I20260929 20:22:48.106591 17621 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
/tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790713339161492-17621-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790713339161492-17621-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790713339161492-17621-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790713339161492-17621-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.53.65:0
--local_ip_for_outbound_sockets=127.17.53.65
--webserver_interface=127.17.53.65
--webserver_port=0
--tserver_master_addrs=127.17.53.126:38699
--keytab_file=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.65.keytab
--principal=kudu/127.17.53.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.53.84:36353
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790713339161492-17621-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790713339161492-17621-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790713339161492-17621-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:22:48.443894 20796 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:22:48.444218 20796 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:22:48.444375 20796 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:22:48.451458 20796 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:22:48.451617 20796 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:22:48.451807 20796 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.53.65
I20260929 20:22:48.461545 20796 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.53.84:36353
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790713339161492-17621-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790713339161492-17621-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.65.keytab
--principal=kudu/127.17.53.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.53.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790713339161492-17621-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.17.53.65
--webserver_port=0
--tserver_master_addrs=127.17.53.126:38699
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.53.65
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790713339161492-17621-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:22:48.462913 20796 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:22:48.464242 20796 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:22:48.477682 20802 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:22:48.477782 20801 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:22:48.477711 20804 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:22:48.478552 20796 server_base.cc:1034] running on GCE node
I20260929 20:22:48.479224 20796 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:22:48.480134 20796 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:22:48.481333 20796 hybrid_clock.cc:648] HybridClock initialized: now 1790713368481263 us; error 66 us; skew 500 ppm
I20260929 20:22:48.481657 20796 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:22:48 dist-test-slave-zbnc krb5kdc[20689](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713368, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.53.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:22:48.484899 20796 init.cc:377] Logged in from keytab as kudu/127.17.53.65@KRBTEST.COM (short username kudu)
I20260929 20:22:48.486434 20796 webserver.cc:460] Webserver started at http://127.17.53.65:41737/ using document root <none> and password file <none>
I20260929 20:22:48.487175 20796 fs_manager.cc:359] Metadata directory not provided
I20260929 20:22:48.487315 20796 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:22:48.487589 20796 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:22:48.491076 20796 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790713339161492-17621-0/minicluster-data/ts-0/data/instance:
uuid: "994b0246d630419f84cca9b3cfd9aaed"
format_stamp: "Formatted at 2026-09-29 20:22:48 on dist-test-slave-zbnc"
server_key: "71c47f666f175963591cfef9a06d7767"
server_key_iv: "bf6bbbad23bf3b87b0a17b9511ba0a0d"
server_key_version: "encryptionkey@0"
I20260929 20:22:48.492216 20796 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790713339161492-17621-0/minicluster-data/ts-0/wal/instance:
uuid: "994b0246d630419f84cca9b3cfd9aaed"
format_stamp: "Formatted at 2026-09-29 20:22:48 on dist-test-slave-zbnc"
server_key: "71c47f666f175963591cfef9a06d7767"
server_key_iv: "bf6bbbad23bf3b87b0a17b9511ba0a0d"
server_key_version: "encryptionkey@0"
I20260929 20:22:48.497826 20796 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 20:22:48.501446 20811 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:22:48.503031 20796 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260929 20:22:48.503225 20796 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790713339161492-17621-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790713339161492-17621-0/minicluster-data/ts-0/wal
uuid: "994b0246d630419f84cca9b3cfd9aaed"
format_stamp: "Formatted at 2026-09-29 20:22:48 on dist-test-slave-zbnc"
server_key: "71c47f666f175963591cfef9a06d7767"
server_key_iv: "bf6bbbad23bf3b87b0a17b9511ba0a0d"
server_key_version: "encryptionkey@0"
I20260929 20:22:48.503428 20796 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790713339161492-17621-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790713339161492-17621-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790713339161492-17621-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:22:48.522524 20796 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:22:48.526959 20796 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:22:48.527262 20796 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:22:48.528224 20796 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:22:48.529572 20796 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:22:48.529680 20796 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:22:48.529834 20796 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:22:48.529909 20796 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:22:48.566274 20796 rpc_server.cc:307] RPC server started. Bound to: 127.17.53.65:43149
I20260929 20:22:48.566314 20924 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.53.65:43149 every 8 connection(s)
I20260929 20:22:48.567631 20796 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790713339161492-17621-0/minicluster-data/ts-0/data/info.pb
I20260929 20:22:48.577567 17621 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu as pid 20796
I20260929 20:22:48.577777 17621 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790713339161492-17621-0/minicluster-data/ts-0/wal/instance
I20260929 20:22:48.578446 17621 external_mini_cluster.cc:1428] Setting key 5bee554c453d73497336d4d38a475d4d
Sep 29 20:22:48 dist-test-slave-zbnc krb5kdc[20689](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713368, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.53.65@KRBTEST.COM for kudu/127.17.53.126@KRBTEST.COM
I20260929 20:22:48.588061 20927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:48.570177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.53.65:59911 (local address 127.17.53.126:38699)
0929 20:22:48.570595 (+ 418us) server_negotiation.cc:206] Beginning negotiation
0929 20:22:48.570600 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:22:48.571666 (+ 1066us) server_negotiation.cc:407] Connection header received
0929 20:22:48.573482 (+ 1816us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:22:48.573488 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:22:48.573580 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:22:48.573706 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:22:48.575418 (+ 1712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:48.576296 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:48.577957 (+ 1661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:48.578359 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:48.578507 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:22:48.582055 (+ 3548us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:22:48.582067 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:22:48.582070 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:22:48.582106 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:22:48.584744 (+ 2638us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:22:48.585480 (+ 736us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:22:48.585486 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:22:48.585491 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:22:48.585611 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:22:48.586115 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:22:48.586121 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:22:48.586124 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:22:48.586387 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:22:48.586500 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0929 20:22:48.587475 (+ 975us) server_negotiation.cc:299] Negotiation successful
0929 20:22:48.587722 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":319,"thread_start_us":217,"threads_started":1}
I20260929 20:22:48.589453 20928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:48.570754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:38699 (local address 127.17.53.65:59911)
0929 20:22:48.571499 (+ 745us) negotiation.cc:107] Waiting for socket to connect
0929 20:22:48.571536 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0929 20:22:48.573193 (+ 1657us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:22:48.573956 (+ 763us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:22:48.573964 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:22:48.574457 (+ 493us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:22:48.575254 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:22:48.575281 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:22:48.576459 (+ 1178us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:22:48.576464 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:22:48.577801 (+ 1337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:22:48.577812 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:22:48.578619 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:22:48.578626 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:22:48.578919 (+ 293us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:22:48.579670 (+ 751us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:22:48.579698 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:22:48.581832 (+ 2134us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:22:48.584892 (+ 3060us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:22:48.584902 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:22:48.584920 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:22:48.585348 (+ 428us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:22:48.585726 (+ 378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:22:48.585733 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:22:48.585740 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:22:48.585981 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:22:48.586518 (+ 537us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:22:48.586529 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:22:48.587063 (+ 534us) client_negotiation.cc:769] Sending connection context
0929 20:22:48.587379 (+ 316us) client_negotiation.cc:240] Negotiation successful
0929 20:22:48.587975 (+ 596us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":520,"thread_start_us":267,"threads_started":1}
I20260929 20:22:48.591071 20925 heartbeater.cc:347] Connected to a master server at 127.17.53.126:38699
I20260929 20:22:48.591369 20925 heartbeater.cc:464] Registering TS with master...
I20260929 20:22:48.592108 20925 heartbeater.cc:511] Master 127.17.53.126:38699 requested a full tablet report, sending...
I20260929 20:22:48.594419 20734 ts_manager.cc:194] Registered new tserver with Master: 994b0246d630419f84cca9b3cfd9aaed (127.17.53.65:43149)
I20260929 20:22:48.595924 20734 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.53.65@KRBTEST.COM'} at 127.17.53.65:59911
WARNING: no policy specified for kudu/127.17.53.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.53.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.53.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.66.keytab.
Entry for principal kudu/127.17.53.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.66.keytab.
I20260929 20:22:48.625947 17621 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
/tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790713339161492-17621-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790713339161492-17621-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790713339161492-17621-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790713339161492-17621-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.53.66:0
--local_ip_for_outbound_sockets=127.17.53.66
--webserver_interface=127.17.53.66
--webserver_port=0
--tserver_master_addrs=127.17.53.126:38699
--keytab_file=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.66.keytab
--principal=kudu/127.17.53.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.53.84:36353
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790713339161492-17621-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790713339161492-17621-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790713339161492-17621-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:22:48.952971 20931 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:22:48.953341 20931 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:22:48.953519 20931 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:22:48.961205 20931 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:22:48.961357 20931 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:22:48.961546 20931 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.53.66
I20260929 20:22:48.971536 20931 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.53.84:36353
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790713339161492-17621-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790713339161492-17621-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.66.keytab
--principal=kudu/127.17.53.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.53.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790713339161492-17621-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.17.53.66
--webserver_port=0
--tserver_master_addrs=127.17.53.126:38699
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.53.66
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790713339161492-17621-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:22:48.972939 20931 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:22:48.974329 20931 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:22:48.987771 20936 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:22:48.987871 20939 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:22:48.987892 20937 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:22:48.989001 20931 server_base.cc:1034] running on GCE node
I20260929 20:22:48.989697 20931 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:22:48.990631 20931 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:22:48.991868 20931 hybrid_clock.cc:648] HybridClock initialized: now 1790713368991805 us; error 56 us; skew 500 ppm
I20260929 20:22:48.992221 20931 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:22:48 dist-test-slave-zbnc krb5kdc[20689](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713368, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.53.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:22:48.995457 20931 init.cc:377] Logged in from keytab as kudu/127.17.53.66@KRBTEST.COM (short username kudu)
I20260929 20:22:48.996969 20931 webserver.cc:460] Webserver started at http://127.17.53.66:35689/ using document root <none> and password file <none>
I20260929 20:22:48.997637 20931 fs_manager.cc:359] Metadata directory not provided
I20260929 20:22:48.997735 20931 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:22:48.997979 20931 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:22:49.000957 20931 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790713339161492-17621-0/minicluster-data/ts-1/data/instance:
uuid: "804d86ee5d8c43dbb0506523291780c7"
format_stamp: "Formatted at 2026-09-29 20:22:48 on dist-test-slave-zbnc"
server_key: "9a129a2d4db70946291c13267dd8e8a2"
server_key_iv: "c85073e59dbfaea412a221f9c3ffe496"
server_key_version: "encryptionkey@0"
I20260929 20:22:49.001770 20931 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790713339161492-17621-0/minicluster-data/ts-1/wal/instance:
uuid: "804d86ee5d8c43dbb0506523291780c7"
format_stamp: "Formatted at 2026-09-29 20:22:48 on dist-test-slave-zbnc"
server_key: "9a129a2d4db70946291c13267dd8e8a2"
server_key_iv: "c85073e59dbfaea412a221f9c3ffe496"
server_key_version: "encryptionkey@0"
I20260929 20:22:49.007597 20931 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260929 20:22:49.011533 20946 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:22:49.013057 20931 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 20:22:49.013276 20931 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790713339161492-17621-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790713339161492-17621-0/minicluster-data/ts-1/wal
uuid: "804d86ee5d8c43dbb0506523291780c7"
format_stamp: "Formatted at 2026-09-29 20:22:48 on dist-test-slave-zbnc"
server_key: "9a129a2d4db70946291c13267dd8e8a2"
server_key_iv: "c85073e59dbfaea412a221f9c3ffe496"
server_key_version: "encryptionkey@0"
I20260929 20:22:49.013477 20931 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790713339161492-17621-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790713339161492-17621-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790713339161492-17621-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:22:49.041571 20931 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:22:49.046195 20931 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:22:49.046515 20931 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:22:49.047602 20931 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:22:49.048961 20931 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:22:49.049039 20931 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:22:49.049180 20931 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:22:49.049233 20931 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:22:49.085852 20931 rpc_server.cc:307] RPC server started. Bound to: 127.17.53.66:43971
I20260929 20:22:49.085947 21059 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.53.66:43971 every 8 connection(s)
I20260929 20:22:49.087339 20931 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790713339161492-17621-0/minicluster-data/ts-1/data/info.pb
I20260929 20:22:49.095122 17621 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu as pid 20931
I20260929 20:22:49.095309 17621 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790713339161492-17621-0/minicluster-data/ts-1/wal/instance
I20260929 20:22:49.095857 17621 external_mini_cluster.cc:1428] Setting key b038b007679d236c0336390c57f2c288
Sep 29 20:22:49 dist-test-slave-zbnc krb5kdc[20689](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713368, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.53.66@KRBTEST.COM for kudu/127.17.53.126@KRBTEST.COM
I20260929 20:22:49.110514 21062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:49.090181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.53.66:51825 (local address 127.17.53.126:38699)
0929 20:22:49.090673 (+ 492us) server_negotiation.cc:206] Beginning negotiation
0929 20:22:49.090680 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:22:49.091704 (+ 1024us) server_negotiation.cc:407] Connection header received
0929 20:22:49.093726 (+ 2022us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:22:49.093730 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:22:49.093795 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:22:49.093889 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:22:49.095809 (+ 1920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:49.096759 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:49.098710 (+ 1951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:49.099219 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:49.099324 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:22:49.103986 (+ 4662us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:22:49.103992 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:22:49.103995 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:22:49.104032 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:22:49.106681 (+ 2649us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:22:49.107534 (+ 853us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:22:49.107542 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:22:49.107548 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:22:49.107734 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:22:49.108395 (+ 661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:22:49.108404 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:22:49.108410 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:22:49.108803 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:22:49.108956 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0929 20:22:49.109744 (+ 788us) server_negotiation.cc:299] Negotiation successful
0929 20:22:49.110013 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":359,"thread_start_us":260,"threads_started":1}
I20260929 20:22:49.111266 21063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:49.090714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:38699 (local address 127.17.53.66:51825)
0929 20:22:49.091463 (+ 749us) negotiation.cc:107] Waiting for socket to connect
0929 20:22:49.091507 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 20:22:49.093432 (+ 1925us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:22:49.094101 (+ 669us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:22:49.094109 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:22:49.094704 (+ 595us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:22:49.095594 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:22:49.095623 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:22:49.096920 (+ 1297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:22:49.096928 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:22:49.098532 (+ 1604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:22:49.098552 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:22:49.099456 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:22:49.099466 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:22:49.099825 (+ 359us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:22:49.100670 (+ 845us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:22:49.100703 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:22:49.103726 (+ 3023us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:22:49.106840 (+ 3114us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:22:49.106850 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:22:49.106905 (+ 55us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:22:49.107357 (+ 452us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:22:49.107942 (+ 585us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:22:49.107946 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:22:49.107948 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:22:49.108243 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:22:49.108939 (+ 696us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:22:49.108946 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:22:49.109307 (+ 361us) client_negotiation.cc:769] Sending connection context
0929 20:22:49.109633 (+ 326us) client_negotiation.cc:240] Negotiation successful
0929 20:22:49.110066 (+ 433us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":510,"thread_start_us":310,"threads_started":1}
I20260929 20:22:49.113013 21060 heartbeater.cc:347] Connected to a master server at 127.17.53.126:38699
I20260929 20:22:49.113404 21060 heartbeater.cc:464] Registering TS with master...
I20260929 20:22:49.114390 21060 heartbeater.cc:511] Master 127.17.53.126:38699 requested a full tablet report, sending...
I20260929 20:22:49.116137 20734 ts_manager.cc:194] Registered new tserver with Master: 804d86ee5d8c43dbb0506523291780c7 (127.17.53.66:43971)
WARNING: no policy specified for kudu/127.17.53.67@KRBTEST.COM; defaulting to no policy
I20260929 20:22:49.117146 20734 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.53.66@KRBTEST.COM'} at 127.17.53.66:51825
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.53.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.53.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.67.keytab.
Entry for principal kudu/127.17.53.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.67.keytab.
I20260929 20:22:49.143198 17621 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
/tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790713339161492-17621-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790713339161492-17621-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790713339161492-17621-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790713339161492-17621-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.53.67:0
--local_ip_for_outbound_sockets=127.17.53.67
--webserver_interface=127.17.53.67
--webserver_port=0
--tserver_master_addrs=127.17.53.126:38699
--keytab_file=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.67.keytab
--principal=kudu/127.17.53.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.53.84:36353
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790713339161492-17621-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790713339161492-17621-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790713339161492-17621-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:22:49.474779 21066 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:22:49.475172 21066 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:22:49.475339 21066 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:22:49.482788 21066 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:22:49.482975 21066 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:22:49.483152 21066 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.53.67
I20260929 20:22:49.493755 21066 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.53.84:36353
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790713339161492-17621-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790713339161492-17621-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.67.keytab
--principal=kudu/127.17.53.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.53.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790713339161492-17621-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.17.53.67
--webserver_port=0
--tserver_master_addrs=127.17.53.126:38699
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.53.67
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790713339161492-17621-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:22:49.495595 21066 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:22:49.496999 21066 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:22:49.511159 21071 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:22:49.511132 21072 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:22:49.511224 21074 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:22:49.511937 21066 server_base.cc:1034] running on GCE node
I20260929 20:22:49.512601 21066 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:22:49.513644 21066 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:22:49.514941 21066 hybrid_clock.cc:648] HybridClock initialized: now 1790713369514866 us; error 65 us; skew 500 ppm
I20260929 20:22:49.515308 21066 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:22:49 dist-test-slave-zbnc krb5kdc[20689](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713369, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.53.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:22:49.518704 21066 init.cc:377] Logged in from keytab as kudu/127.17.53.67@KRBTEST.COM (short username kudu)
I20260929 20:22:49.520383 21066 webserver.cc:460] Webserver started at http://127.17.53.67:33013/ using document root <none> and password file <none>
I20260929 20:22:49.521103 21066 fs_manager.cc:359] Metadata directory not provided
I20260929 20:22:49.521252 21066 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:22:49.521572 21066 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:22:49.524544 21066 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790713339161492-17621-0/minicluster-data/ts-2/data/instance:
uuid: "ce9b33b914264c71a70c049b04aa7289"
format_stamp: "Formatted at 2026-09-29 20:22:49 on dist-test-slave-zbnc"
server_key: "d5cb8aae144900460784b90a029ded1a"
server_key_iv: "d6b8448d04cebcce8318636a01f7f087"
server_key_version: "encryptionkey@0"
I20260929 20:22:49.525446 21066 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790713339161492-17621-0/minicluster-data/ts-2/wal/instance:
uuid: "ce9b33b914264c71a70c049b04aa7289"
format_stamp: "Formatted at 2026-09-29 20:22:49 on dist-test-slave-zbnc"
server_key: "d5cb8aae144900460784b90a029ded1a"
server_key_iv: "d6b8448d04cebcce8318636a01f7f087"
server_key_version: "encryptionkey@0"
I20260929 20:22:49.530526 21066 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260929 20:22:49.534152 21081 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:22:49.535567 21066 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 20:22:49.535768 21066 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790713339161492-17621-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790713339161492-17621-0/minicluster-data/ts-2/wal
uuid: "ce9b33b914264c71a70c049b04aa7289"
format_stamp: "Formatted at 2026-09-29 20:22:49 on dist-test-slave-zbnc"
server_key: "d5cb8aae144900460784b90a029ded1a"
server_key_iv: "d6b8448d04cebcce8318636a01f7f087"
server_key_version: "encryptionkey@0"
I20260929 20:22:49.535964 21066 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790713339161492-17621-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790713339161492-17621-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790713339161492-17621-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:22:49.555569 21066 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:22:49.559937 21066 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:22:49.560245 21066 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:22:49.561354 21066 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:22:49.562703 21066 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:22:49.562819 21066 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:22:49.562988 21066 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:22:49.563072 21066 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:22:49.598896 20925 heartbeater.cc:503] Master 127.17.53.126:38699 was elected leader, sending a full tablet report...
I20260929 20:22:49.599786 21066 rpc_server.cc:307] RPC server started. Bound to: 127.17.53.67:41283
I20260929 20:22:49.599821 21194 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.53.67:41283 every 8 connection(s)
I20260929 20:22:49.601038 21066 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790713339161492-17621-0/minicluster-data/ts-2/data/info.pb
I20260929 20:22:49.602108 17621 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu as pid 21066
I20260929 20:22:49.602294 17621 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790713339161492-17621-0/minicluster-data/ts-2/wal/instance
I20260929 20:22:49.602895 17621 external_mini_cluster.cc:1428] Setting key ffe1a0843e632a6c2dae932028b7c730
Sep 29 20:22:49 dist-test-slave-zbnc krb5kdc[20689](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713369, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.53.67@KRBTEST.COM for kudu/127.17.53.126@KRBTEST.COM
I20260929 20:22:49.621644 21062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:49.603998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.53.67:47193 (local address 127.17.53.126:38699)
0929 20:22:49.604169 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0929 20:22:49.604173 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:22:49.605553 (+ 1380us) server_negotiation.cc:407] Connection header received
0929 20:22:49.607646 (+ 2093us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:22:49.607651 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:22:49.607729 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:22:49.607817 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:22:49.609286 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:49.610010 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:49.611633 (+ 1623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:49.612044 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:49.612197 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:22:49.615619 (+ 3422us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:22:49.615624 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:22:49.615627 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:22:49.615658 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:22:49.618676 (+ 3018us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:22:49.619425 (+ 749us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:22:49.619430 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:22:49.619434 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:22:49.619551 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:22:49.619984 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:22:49.619988 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:22:49.619992 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:22:49.620249 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:22:49.620422 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0929 20:22:49.621070 (+ 648us) server_negotiation.cc:299] Negotiation successful
0929 20:22:49.621288 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 20:22:49.622726 21198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:49.604571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:38699 (local address 127.17.53.67:47193)
0929 20:22:49.605403 (+ 832us) negotiation.cc:107] Waiting for socket to connect
0929 20:22:49.605446 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 20:22:49.607391 (+ 1945us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:22:49.608079 (+ 688us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:22:49.608087 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:22:49.608570 (+ 483us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:22:49.609127 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:22:49.609144 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:22:49.610156 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:22:49.610163 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:22:49.611490 (+ 1327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:22:49.611502 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:22:49.612286 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:22:49.612292 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:22:49.612517 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:22:49.613127 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:22:49.613153 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:22:49.615385 (+ 2232us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:22:49.618813 (+ 3428us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:22:49.618821 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:22:49.618839 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:22:49.619299 (+ 460us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:22:49.619678 (+ 379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:22:49.619682 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:22:49.619686 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:22:49.619865 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:22:49.620377 (+ 512us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:22:49.620385 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:22:49.620725 (+ 340us) client_negotiation.cc:769] Sending connection context
0929 20:22:49.620993 (+ 268us) client_negotiation.cc:240] Negotiation successful
0929 20:22:49.621534 (+ 541us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":604,"thread_start_us":388,"threads_started":1}
I20260929 20:22:49.624112 21195 heartbeater.cc:347] Connected to a master server at 127.17.53.126:38699
I20260929 20:22:49.624413 21195 heartbeater.cc:464] Registering TS with master...
I20260929 20:22:49.625262 21195 heartbeater.cc:511] Master 127.17.53.126:38699 requested a full tablet report, sending...
I20260929 20:22:49.626633 20734 ts_manager.cc:194] Registered new tserver with Master: ce9b33b914264c71a70c049b04aa7289 (127.17.53.67:41283)
I20260929 20:22:49.627502 20734 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.53.67@KRBTEST.COM'} at 127.17.53.67:47193
I20260929 20:22:49.631017 17621 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 29 20:22:49 dist-test-slave-zbnc krb5kdc[20689](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713369, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 29 20:22:49 dist-test-slave-zbnc krb5kdc[20689](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713369, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.17.53.126@KRBTEST.COM
I20260929 20:22:49.679352 21062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:49.664767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41820 (local address 127.17.53.126:38699)
0929 20:22:49.664966 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0929 20:22:49.664974 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 20:22:49.665403 (+ 429us) server_negotiation.cc:407] Connection header received
0929 20:22:49.665627 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:22:49.665631 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:22:49.665704 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:22:49.665813 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:22:49.667217 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:49.667998 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:49.669263 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:49.669693 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:49.669837 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:22:49.674306 (+ 4469us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:22:49.674311 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:22:49.674314 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:22:49.674349 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:22:49.677009 (+ 2660us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:22:49.677636 (+ 627us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:22:49.677640 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:22:49.677642 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:22:49.677759 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:22:49.678109 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:22:49.678112 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:22:49.678114 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:22:49.678413 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:22:49.678654 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0929 20:22:49.678965 (+ 311us) server_negotiation.cc:299] Negotiation successful
0929 20:22:49.679116 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
W20260929 20:22:49.679939 20734 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:41820
I20260929 20:22:49.682008 17621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu pid 20796
Sep 29 20:22:49 dist-test-slave-zbnc krb5kdc[20689](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713369, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.17.53.65@KRBTEST.COM
I20260929 20:22:49.698284 21207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:49.683101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55928 (local address 127.17.53.65:43149)
0929 20:22:49.683651 (+ 550us) server_negotiation.cc:206] Beginning negotiation
0929 20:22:49.683658 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:22:49.683682 (+ 24us) server_negotiation.cc:407] Connection header received
0929 20:22:49.683735 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:22:49.683741 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:22:49.683946 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:22:49.684068 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:22:49.685212 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:49.686220 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:49.687468 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:49.688129 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:49.688272 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:22:49.692206 (+ 3934us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:22:49.692218 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:22:49.692234 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:22:49.692269 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:22:49.695671 (+ 3402us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:22:49.696358 (+ 687us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:22:49.696366 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:22:49.696368 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:22:49.696488 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:22:49.696820 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:22:49.696823 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:22:49.696825 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:22:49.697152 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:22:49.697367 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0929 20:22:49.697643 (+ 276us) server_negotiation.cc:299] Negotiation successful
0929 20:22:49.697851 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":405,"thread_start_us":267,"threads_started":1}
W20260929 20:22:49.699832 20920 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:55928
W20260929 20:22:49.701050 17621 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu pid 20796: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 20:22:49.701268 17621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu with pid 20796
I20260929 20:22:49.717170 17621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu pid 20931
Sep 29 20:22:49 dist-test-slave-zbnc krb5kdc[20689](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713369, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.17.53.66@KRBTEST.COM
I20260929 20:22:49.734301 21208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:49.718291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32904 (local address 127.17.53.66:43971)
0929 20:22:49.718968 (+ 677us) server_negotiation.cc:206] Beginning negotiation
0929 20:22:49.718975 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:22:49.719002 (+ 27us) server_negotiation.cc:407] Connection header received
0929 20:22:49.719078 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:22:49.719085 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:22:49.719298 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:22:49.719428 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:22:49.720687 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:49.721724 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:49.723128 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:49.723690 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:49.723800 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:22:49.728522 (+ 4722us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:22:49.728533 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:22:49.728549 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:22:49.728581 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:22:49.731462 (+ 2881us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:22:49.732059 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:22:49.732065 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:22:49.732067 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:22:49.732203 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:22:49.732632 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:22:49.732635 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:22:49.732637 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:22:49.732960 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:22:49.733210 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0929 20:22:49.733573 (+ 363us) server_negotiation.cc:299] Negotiation successful
0929 20:22:49.733826 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":472,"thread_start_us":343,"threads_started":1}
W20260929 20:22:49.735687 21055 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:32904
W20260929 20:22:49.736899 17621 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu pid 20931: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 20:22:49.737079 17621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu with pid 20931
I20260929 20:22:49.752995 17621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu pid 21066
Sep 29 20:22:49 dist-test-slave-zbnc krb5kdc[20689](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713369, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.17.53.67@KRBTEST.COM
I20260929 20:22:49.770439 21209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:49.754204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41634 (local address 127.17.53.67:41283)
0929 20:22:49.754925 (+ 721us) server_negotiation.cc:206] Beginning negotiation
0929 20:22:49.754938 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 20:22:49.754968 (+ 30us) server_negotiation.cc:407] Connection header received
0929 20:22:49.755035 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:22:49.755042 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:22:49.755365 (+ 323us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:22:49.755587 (+ 222us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:22:49.757275 (+ 1688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:49.758327 (+ 1052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:49.759688 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:49.760263 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:49.760419 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:22:49.764799 (+ 4380us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:22:49.764810 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:22:49.764826 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:22:49.764863 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:22:49.767578 (+ 2715us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:22:49.768287 (+ 709us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:22:49.768294 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:22:49.768297 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:22:49.768417 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:22:49.768789 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:22:49.768795 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:22:49.768798 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:22:49.769153 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:22:49.769460 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0929 20:22:49.769805 (+ 345us) server_negotiation.cc:299] Negotiation successful
0929 20:22:49.770092 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":522,"thread_start_us":370,"threads_started":1}
W20260929 20:22:49.771867 21190 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:41634
W20260929 20:22:49.772943 17621 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu pid 21066: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 20:22:49.773104 17621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu with pid 21066
I20260929 20:22:49.789286 17621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu pid 20702
I20260929 20:22:49.790491 20765 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:22:49.891985 17621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu with pid 20702
2026-09-29T20:22:49Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (3655 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790713339161492-17621-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 20:22:49 dist-test-slave-zbnc krb5kdc[21213](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 20:22:49 dist-test-slave-zbnc krb5kdc[21213](info): set up 2 sockets
Sep 29 20:22:49 dist-test-slave-zbnc krb5kdc[21213](info): commencing operation
krb5kdc: starting...
W20260929 20:22:50.988524 17621 mini_kdc.cc:121] Time spent starting KDC: real 1.054s user 0.004s sys 0.012s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790713339161492-17621-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790713339161492-17621-0/krb5kdc/test-user.keytab.
Sep 29 20:22:51 dist-test-slave-zbnc krb5kdc[21213](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713371, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T20:22:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T20:22:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.17.53.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.53.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.53.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.126.keytab.
Entry for principal kudu/127.17.53.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.126.keytab.
I20260929 20:22:51.210605 17621 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
/tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790713339161492-17621-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790713339161492-17621-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790713339161492-17621-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790713339161492-17621-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.53.126:39531
--webserver_interface=127.17.53.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.126.keytab
--principal=kudu/127.17.53.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.53.84:42555
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.53.126:39531
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790713339161492-17621-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790713339161492-17621-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790713339161492-17621-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:22:51.538096 21226 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:22:51.538388 21226 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:22:51.538478 21226 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:22:51.545897 21226 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:22:51.546005 21226 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:22:51.546051 21226 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:22:51.546108 21226 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:22:51.546144 21226 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:22:51.556109 21226 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.53.84:42555
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790713339161492-17621-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790713339161492-17621-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.53.126:39531
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.126.keytab
--principal=kudu/127.17.53.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.53.126:39531
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790713339161492-17621-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.53.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790713339161492-17621-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:22:51.557401 21226 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:22:51.558656 21226 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:22:51.570734 21233 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:22:51.570688 21235 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:22:51.570756 21232 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:22:51.571589 21226 server_base.cc:1034] running on GCE node
I20260929 20:22:51.572410 21226 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:22:51.573827 21226 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:22:51.575018 21226 hybrid_clock.cc:648] HybridClock initialized: now 1790713371574953 us; error 52 us; skew 500 ppm
I20260929 20:22:51.575378 21226 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:22:51 dist-test-slave-zbnc krb5kdc[21213](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713371, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.53.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:22:51.578836 21226 init.cc:377] Logged in from keytab as kudu/127.17.53.126@KRBTEST.COM (short username kudu)
I20260929 20:22:51.580415 21226 webserver.cc:460] Webserver started at http://127.17.53.126:33273/ using document root <none> and password file <none>
I20260929 20:22:51.581146 21226 fs_manager.cc:359] Metadata directory not provided
I20260929 20:22:51.581277 21226 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:22:51.581610 21226 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:22:51.584673 21226 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790713339161492-17621-0/minicluster-data/master-0/data/instance:
uuid: "fdea247c5ff04dc0a6a1f4ca5a12a667"
format_stamp: "Formatted at 2026-09-29 20:22:51 on dist-test-slave-zbnc"
server_key: "4d1a24679600f0b0dea4ef1a04f607da"
server_key_iv: "964170d0a0cf08ce7179ba1918faa6bc"
server_key_version: "encryptionkey@0"
I20260929 20:22:51.585520 21226 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790713339161492-17621-0/minicluster-data/master-0/wal/instance:
uuid: "fdea247c5ff04dc0a6a1f4ca5a12a667"
format_stamp: "Formatted at 2026-09-29 20:22:51 on dist-test-slave-zbnc"
server_key: "4d1a24679600f0b0dea4ef1a04f607da"
server_key_iv: "964170d0a0cf08ce7179ba1918faa6bc"
server_key_version: "encryptionkey@0"
I20260929 20:22:51.590821 21226 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.002s
I20260929 20:22:51.594616 21242 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:22:51.596189 21226 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 20:22:51.596385 21226 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790713339161492-17621-0/minicluster-data/master-0/data,/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790713339161492-17621-0/minicluster-data/master-0/wal
uuid: "fdea247c5ff04dc0a6a1f4ca5a12a667"
format_stamp: "Formatted at 2026-09-29 20:22:51 on dist-test-slave-zbnc"
server_key: "4d1a24679600f0b0dea4ef1a04f607da"
server_key_iv: "964170d0a0cf08ce7179ba1918faa6bc"
server_key_version: "encryptionkey@0"
I20260929 20:22:51.596644 21226 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790713339161492-17621-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790713339161492-17621-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790713339161492-17621-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:22:51.612713 21226 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:22:51.616788 21226 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:22:51.617084 21226 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:22:51.636019 21226 rpc_server.cc:307] RPC server started. Bound to: 127.17.53.126:39531
I20260929 20:22:51.636026 21293 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.53.126:39531 every 8 connection(s)
I20260929 20:22:51.637509 21226 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790713339161492-17621-0/minicluster-data/master-0/data/info.pb
I20260929 20:22:51.639313 17621 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu as pid 21226
I20260929 20:22:51.639568 17621 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790713339161492-17621-0/minicluster-data/master-0/wal/instance
I20260929 20:22:51.640499 17621 external_mini_cluster.cc:1428] Setting key 67300e4dbc2ada9af48ec5302edc2df0
I20260929 20:22:51.643086 21294 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 29 20:22:51 dist-test-slave-zbnc krb5kdc[21213](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713371, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.53.126@KRBTEST.COM
I20260929 20:22:51.657603 21294 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fdea247c5ff04dc0a6a1f4ca5a12a667: Bootstrap starting.
I20260929 20:22:51.661358 21294 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fdea247c5ff04dc0a6a1f4ca5a12a667: Neither blocks nor log segments found. Creating new log.
I20260929 20:22:51.662535 21294 log.cc:824] T 00000000000000000000000000000000 P fdea247c5ff04dc0a6a1f4ca5a12a667: Log is configured to *not* fsync() on all Append() calls
I20260929 20:22:51.665539 21297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:51.642915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57068 (local address 127.17.53.126:39531)
0929 20:22:51.643745 (+ 830us) server_negotiation.cc:206] Beginning negotiation
0929 20:22:51.643758 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 20:22:51.643800 (+ 42us) server_negotiation.cc:407] Connection header received
0929 20:22:51.645492 (+ 1692us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:22:51.645515 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:22:51.645917 (+ 402us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:22:51.646281 (+ 364us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:22:51.648452 (+ 2171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:51.649848 (+ 1396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:51.651786 (+ 1938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:51.652436 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:51.652563 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:22:51.657550 (+ 4987us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:22:51.657558 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:22:51.657573 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:22:51.657660 (+ 87us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:22:51.660937 (+ 3277us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:22:51.661675 (+ 738us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:22:51.661682 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:22:51.661689 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:22:51.661862 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:22:51.662274 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:22:51.662284 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:22:51.662286 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:22:51.662956 (+ 670us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:22:51.663252 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0929 20:22:51.663731 (+ 479us) server_negotiation.cc:299] Negotiation successful
0929 20:22:51.664211 (+ 480us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":607,"thread_start_us":397,"threads_started":1}
I20260929 20:22:51.666144 21294 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fdea247c5ff04dc0a6a1f4ca5a12a667: No bootstrap required, opened a new log
I20260929 20:22:51.674176 21294 raft_consensus.cc:348] T 00000000000000000000000000000000 P fdea247c5ff04dc0a6a1f4ca5a12a667 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdea247c5ff04dc0a6a1f4ca5a12a667" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 39531 } }
I20260929 20:22:51.674547 21294 raft_consensus.cc:374] T 00000000000000000000000000000000 P fdea247c5ff04dc0a6a1f4ca5a12a667 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:22:51.674623 21294 raft_consensus.cc:729] T 00000000000000000000000000000000 P fdea247c5ff04dc0a6a1f4ca5a12a667 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fdea247c5ff04dc0a6a1f4ca5a12a667, State: Initialized, Role: FOLLOWER
I20260929 20:22:51.675222 21294 consensus_queue.cc:260] T 00000000000000000000000000000000 P fdea247c5ff04dc0a6a1f4ca5a12a667 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdea247c5ff04dc0a6a1f4ca5a12a667" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 39531 } }
I20260929 20:22:51.675429 21294 raft_consensus.cc:388] T 00000000000000000000000000000000 P fdea247c5ff04dc0a6a1f4ca5a12a667 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 20:22:51.675529 21294 raft_consensus.cc:482] T 00000000000000000000000000000000 P fdea247c5ff04dc0a6a1f4ca5a12a667 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 20:22:51.675654 21294 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fdea247c5ff04dc0a6a1f4ca5a12a667 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:22:51.677832 21294 raft_consensus.cc:504] T 00000000000000000000000000000000 P fdea247c5ff04dc0a6a1f4ca5a12a667 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdea247c5ff04dc0a6a1f4ca5a12a667" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 39531 } }
I20260929 20:22:51.678330 21294 leader_election.cc:302] T 00000000000000000000000000000000 P fdea247c5ff04dc0a6a1f4ca5a12a667 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fdea247c5ff04dc0a6a1f4ca5a12a667; no voters:
I20260929 20:22:51.679141 21294 leader_election.cc:288] T 00000000000000000000000000000000 P fdea247c5ff04dc0a6a1f4ca5a12a667 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 20:22:51.680377 21299 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fdea247c5ff04dc0a6a1f4ca5a12a667 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:22:51.681401 21299 raft_consensus.cc:686] T 00000000000000000000000000000000 P fdea247c5ff04dc0a6a1f4ca5a12a667 [term 1 LEADER]: Becoming Leader. State: Replica: fdea247c5ff04dc0a6a1f4ca5a12a667, State: Running, Role: LEADER
I20260929 20:22:51.681946 21299 consensus_queue.cc:237] T 00000000000000000000000000000000 P fdea247c5ff04dc0a6a1f4ca5a12a667 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdea247c5ff04dc0a6a1f4ca5a12a667" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 39531 } }
I20260929 20:22:51.682485 21294 sys_catalog.cc:564] T 00000000000000000000000000000000 P fdea247c5ff04dc0a6a1f4ca5a12a667 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:22:51.684985 21300 sys_catalog.cc:455] T 00000000000000000000000000000000 P fdea247c5ff04dc0a6a1f4ca5a12a667 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fdea247c5ff04dc0a6a1f4ca5a12a667" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdea247c5ff04dc0a6a1f4ca5a12a667" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 39531 } } }
I20260929 20:22:51.685520 21300 sys_catalog.cc:458] T 00000000000000000000000000000000 P fdea247c5ff04dc0a6a1f4ca5a12a667 [sys.catalog]: This master's current role is: LEADER
I20260929 20:22:51.685015 21301 sys_catalog.cc:455] T 00000000000000000000000000000000 P fdea247c5ff04dc0a6a1f4ca5a12a667 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fdea247c5ff04dc0a6a1f4ca5a12a667. Latest consensus state: current_term: 1 leader_uuid: "fdea247c5ff04dc0a6a1f4ca5a12a667" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fdea247c5ff04dc0a6a1f4ca5a12a667" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 39531 } } }
I20260929 20:22:51.686154 21301 sys_catalog.cc:458] T 00000000000000000000000000000000 P fdea247c5ff04dc0a6a1f4ca5a12a667 [sys.catalog]: This master's current role is: LEADER
I20260929 20:22:51.686939 21307 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:22:51.692605 21307 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:22:51.709342 21307 catalog_manager.cc:1347] Generated new cluster ID: b4280c81acc2441d834968ef963ce402
I20260929 20:22:51.709473 21307 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:22:51.725596 21307 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 20:22:51.727028 21307 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:22:51.738325 21307 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fdea247c5ff04dc0a6a1f4ca5a12a667: Generated new TSK 0
I20260929 20:22:51.739351 21307 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.17.53.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.53.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.53.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.65.keytab.
Entry for principal kudu/127.17.53.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.65.keytab.
I20260929 20:22:51.795346 17621 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
/tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790713339161492-17621-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790713339161492-17621-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790713339161492-17621-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790713339161492-17621-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.53.65:0
--local_ip_for_outbound_sockets=127.17.53.65
--webserver_interface=127.17.53.65
--webserver_port=0
--tserver_master_addrs=127.17.53.126:39531
--keytab_file=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.65.keytab
--principal=kudu/127.17.53.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.53.84:42555
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790713339161492-17621-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790713339161492-17621-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790713339161492-17621-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:22:52.133591 21320 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:22:52.133894 21320 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:22:52.134094 21320 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:22:52.141556 21320 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:22:52.141685 21320 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:22:52.141889 21320 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.53.65
I20260929 20:22:52.151957 21320 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.53.84:42555
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790713339161492-17621-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790713339161492-17621-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.65.keytab
--principal=kudu/127.17.53.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.53.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790713339161492-17621-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.17.53.65
--webserver_port=0
--tserver_master_addrs=127.17.53.126:39531
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.53.65
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790713339161492-17621-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:22:52.153405 21320 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:22:52.154807 21320 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:22:52.169447 21326 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:22:52.169441 21328 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:22:52.169727 21320 server_base.cc:1034] running on GCE node
W20260929 20:22:52.170075 21325 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:22:52.170758 21320 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:22:52.171770 21320 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:22:52.172973 21320 hybrid_clock.cc:648] HybridClock initialized: now 1790713372172919 us; error 43 us; skew 500 ppm
I20260929 20:22:52.173336 21320 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:22:52 dist-test-slave-zbnc krb5kdc[21213](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713372, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.53.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:22:52.176561 21320 init.cc:377] Logged in from keytab as kudu/127.17.53.65@KRBTEST.COM (short username kudu)
I20260929 20:22:52.178174 21320 webserver.cc:460] Webserver started at http://127.17.53.65:36489/ using document root <none> and password file <none>
I20260929 20:22:52.179001 21320 fs_manager.cc:359] Metadata directory not provided
I20260929 20:22:52.179176 21320 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:22:52.179509 21320 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:22:52.182582 21320 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790713339161492-17621-0/minicluster-data/ts-0/data/instance:
uuid: "5f29a9839c9b48ddad67fa252e763a0a"
format_stamp: "Formatted at 2026-09-29 20:22:52 on dist-test-slave-zbnc"
server_key: "8dabfcc826ceb2ed3f94afc1d0927424"
server_key_iv: "8f5a902a5209706fdc59b7dd7e442f87"
server_key_version: "encryptionkey@0"
I20260929 20:22:52.183486 21320 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790713339161492-17621-0/minicluster-data/ts-0/wal/instance:
uuid: "5f29a9839c9b48ddad67fa252e763a0a"
format_stamp: "Formatted at 2026-09-29 20:22:52 on dist-test-slave-zbnc"
server_key: "8dabfcc826ceb2ed3f94afc1d0927424"
server_key_iv: "8f5a902a5209706fdc59b7dd7e442f87"
server_key_version: "encryptionkey@0"
I20260929 20:22:52.189119 21320 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.001s
I20260929 20:22:52.192867 21335 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:22:52.194258 21320 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 20:22:52.194479 21320 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790713339161492-17621-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790713339161492-17621-0/minicluster-data/ts-0/wal
uuid: "5f29a9839c9b48ddad67fa252e763a0a"
format_stamp: "Formatted at 2026-09-29 20:22:52 on dist-test-slave-zbnc"
server_key: "8dabfcc826ceb2ed3f94afc1d0927424"
server_key_iv: "8f5a902a5209706fdc59b7dd7e442f87"
server_key_version: "encryptionkey@0"
I20260929 20:22:52.194721 21320 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790713339161492-17621-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790713339161492-17621-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790713339161492-17621-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:22:52.218771 21320 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:22:52.223135 21320 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:22:52.223452 21320 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:22:52.224457 21320 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:22:52.225838 21320 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:22:52.225912 21320 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:22:52.226044 21320 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:22:52.226106 21320 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:22:52.261308 21320 rpc_server.cc:307] RPC server started. Bound to: 127.17.53.65:44091
I20260929 20:22:52.261353 21449 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.53.65:44091 every 8 connection(s)
I20260929 20:22:52.262655 21320 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790713339161492-17621-0/minicluster-data/ts-0/data/info.pb
I20260929 20:22:52.265828 17621 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu as pid 21320
I20260929 20:22:52.266060 17621 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790713339161492-17621-0/minicluster-data/ts-0/wal/instance
I20260929 20:22:52.267053 17621 external_mini_cluster.cc:1428] Setting key a781d6e20ce498c715be85ebfab85e0e
Sep 29 20:22:52 dist-test-slave-zbnc krb5kdc[21213](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713372, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.53.65@KRBTEST.COM for kudu/127.17.53.126@KRBTEST.COM
I20260929 20:22:52.285516 21452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:52.265308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.53.65:38325 (local address 127.17.53.126:39531)
0929 20:22:52.265786 (+ 478us) server_negotiation.cc:206] Beginning negotiation
0929 20:22:52.265790 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:22:52.267101 (+ 1311us) server_negotiation.cc:407] Connection header received
0929 20:22:52.269200 (+ 2099us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:22:52.269205 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:22:52.269278 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:22:52.269419 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:22:52.271002 (+ 1583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:52.272183 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:52.273895 (+ 1712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:52.274430 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:52.274529 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:22:52.279204 (+ 4675us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:22:52.279209 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:22:52.279212 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:22:52.279244 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:22:52.281900 (+ 2656us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:22:52.282960 (+ 1060us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:22:52.282964 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:22:52.282966 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:22:52.283134 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:22:52.283647 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:22:52.283650 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:22:52.283652 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:22:52.283931 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:22:52.284045 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0929 20:22:52.285052 (+ 1007us) server_negotiation.cc:299] Negotiation successful
0929 20:22:52.285240 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":359,"thread_start_us":246,"threads_started":1}
WARNING: no policy specified for kudu/127.17.53.66@KRBTEST.COM; defaulting to no policy
I20260929 20:22:52.287086 21453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:52.265916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:39531 (local address 127.17.53.65:38325)
0929 20:22:52.266925 (+ 1009us) negotiation.cc:107] Waiting for socket to connect
0929 20:22:52.266979 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0929 20:22:52.268879 (+ 1900us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:22:52.269693 (+ 814us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:22:52.269702 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:22:52.270211 (+ 509us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:22:52.270789 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:22:52.270805 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:22:52.272353 (+ 1548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:22:52.272357 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:22:52.273740 (+ 1383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:22:52.273752 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:22:52.274753 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:22:52.274761 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:22:52.275129 (+ 368us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:22:52.275925 (+ 796us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:22:52.275966 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:22:52.278951 (+ 2985us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:22:52.282077 (+ 3126us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:22:52.282087 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:22:52.282106 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:22:52.282812 (+ 706us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:22:52.283229 (+ 417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:22:52.283236 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:22:52.283242 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:22:52.283512 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:22:52.284055 (+ 543us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:22:52.284065 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:22:52.284514 (+ 449us) client_negotiation.cc:769] Sending connection context
0929 20:22:52.284903 (+ 389us) client_negotiation.cc:240] Negotiation successful
0929 20:22:52.285479 (+ 576us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":715,"thread_start_us":399,"threads_started":1}
I20260929 20:22:52.288625 21450 heartbeater.cc:347] Connected to a master server at 127.17.53.126:39531
I20260929 20:22:52.288940 21450 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.53.66@KRBTEST.COM" created.
I20260929 20:22:52.289737 21450 heartbeater.cc:511] Master 127.17.53.126:39531 requested a full tablet report, sending...
I20260929 20:22:52.291934 21259 ts_manager.cc:194] Registered new tserver with Master: 5f29a9839c9b48ddad67fa252e763a0a (127.17.53.65:44091)
I20260929 20:22:52.293404 21259 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.53.65@KRBTEST.COM'} at 127.17.53.65:38325
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.53.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.66.keytab.
Entry for principal kudu/127.17.53.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.66.keytab.
I20260929 20:22:52.315747 17621 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
/tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790713339161492-17621-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790713339161492-17621-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790713339161492-17621-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790713339161492-17621-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.53.66:0
--local_ip_for_outbound_sockets=127.17.53.66
--webserver_interface=127.17.53.66
--webserver_port=0
--tserver_master_addrs=127.17.53.126:39531
--keytab_file=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.66.keytab
--principal=kudu/127.17.53.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.53.84:42555
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790713339161492-17621-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790713339161492-17621-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790713339161492-17621-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:22:52.647601 21456 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:22:52.647884 21456 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:22:52.648056 21456 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:22:52.655568 21456 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:22:52.655680 21456 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:22:52.655891 21456 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.53.66
I20260929 20:22:52.665992 21456 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.53.84:42555
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790713339161492-17621-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790713339161492-17621-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.66.keytab
--principal=kudu/127.17.53.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.53.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790713339161492-17621-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.17.53.66
--webserver_port=0
--tserver_master_addrs=127.17.53.126:39531
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.53.66
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790713339161492-17621-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:22:52.667357 21456 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:22:52.668720 21456 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:22:52.682144 21462 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:22:52.682380 21463 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:22:52.682632 21465 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:22:52.684084 21456 server_base.cc:1034] running on GCE node
I20260929 20:22:52.684830 21456 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:22:52.685859 21456 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:22:52.687114 21456 hybrid_clock.cc:648] HybridClock initialized: now 1790713372687058 us; error 54 us; skew 500 ppm
I20260929 20:22:52.687505 21456 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:22:52 dist-test-slave-zbnc krb5kdc[21213](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713372, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.53.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:22:52.690910 21456 init.cc:377] Logged in from keytab as kudu/127.17.53.66@KRBTEST.COM (short username kudu)
I20260929 20:22:52.692572 21456 webserver.cc:460] Webserver started at http://127.17.53.66:35343/ using document root <none> and password file <none>
I20260929 20:22:52.693313 21456 fs_manager.cc:359] Metadata directory not provided
I20260929 20:22:52.693470 21456 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:22:52.693787 21456 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:22:52.696938 21456 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790713339161492-17621-0/minicluster-data/ts-1/data/instance:
uuid: "1a17b9fa720d42c6a84068519de73682"
format_stamp: "Formatted at 2026-09-29 20:22:52 on dist-test-slave-zbnc"
server_key: "f4ebe69d54dd70ef4c86fb5d41a96bdf"
server_key_iv: "78458fc5d4e15649f62177e3bcfb05e1"
server_key_version: "encryptionkey@0"
I20260929 20:22:52.697757 21456 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790713339161492-17621-0/minicluster-data/ts-1/wal/instance:
uuid: "1a17b9fa720d42c6a84068519de73682"
format_stamp: "Formatted at 2026-09-29 20:22:52 on dist-test-slave-zbnc"
server_key: "f4ebe69d54dd70ef4c86fb5d41a96bdf"
server_key_iv: "78458fc5d4e15649f62177e3bcfb05e1"
server_key_version: "encryptionkey@0"
I20260929 20:22:52.703612 21456 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260929 20:22:52.707361 21472 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:22:52.708966 21456 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 20:22:52.709224 21456 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790713339161492-17621-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790713339161492-17621-0/minicluster-data/ts-1/wal
uuid: "1a17b9fa720d42c6a84068519de73682"
format_stamp: "Formatted at 2026-09-29 20:22:52 on dist-test-slave-zbnc"
server_key: "f4ebe69d54dd70ef4c86fb5d41a96bdf"
server_key_iv: "78458fc5d4e15649f62177e3bcfb05e1"
server_key_version: "encryptionkey@0"
I20260929 20:22:52.709477 21456 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790713339161492-17621-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790713339161492-17621-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790713339161492-17621-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:22:52.735411 21456 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:22:52.739900 21456 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:22:52.740257 21456 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:22:52.741361 21456 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:22:52.742811 21456 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:22:52.742954 21456 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260929 20:22:52.743142 21456 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:22:52.743242 21456 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:22:52.779145 21456 rpc_server.cc:307] RPC server started. Bound to: 127.17.53.66:32915
I20260929 20:22:52.779202 21585 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.53.66:32915 every 8 connection(s)
I20260929 20:22:52.780447 21456 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790713339161492-17621-0/minicluster-data/ts-1/data/info.pb
I20260929 20:22:52.785424 17621 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu as pid 21456
I20260929 20:22:52.785629 17621 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790713339161492-17621-0/minicluster-data/ts-1/wal/instance
I20260929 20:22:52.786054 17621 external_mini_cluster.cc:1428] Setting key dec1ccb77ef75ac566acd1776b8341f5
Sep 29 20:22:52 dist-test-slave-zbnc krb5kdc[21213](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713372, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.53.66@KRBTEST.COM for kudu/127.17.53.126@KRBTEST.COM
I20260929 20:22:52.801143 21452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:52.783152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.53.66:56889 (local address 127.17.53.126:39531)
0929 20:22:52.783320 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0929 20:22:52.783325 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:22:52.784802 (+ 1477us) server_negotiation.cc:407] Connection header received
0929 20:22:52.786777 (+ 1975us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:22:52.786783 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:22:52.786904 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:22:52.787016 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:22:52.788598 (+ 1582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:52.789302 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:52.790888 (+ 1586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:52.791335 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:52.791476 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:22:52.795068 (+ 3592us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:22:52.795072 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:22:52.795076 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:22:52.795107 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:22:52.797650 (+ 2543us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:22:52.798516 (+ 866us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:22:52.798520 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:22:52.798522 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:22:52.798650 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:22:52.799177 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:22:52.799182 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:22:52.799186 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:22:52.799508 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:22:52.799674 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0929 20:22:52.800482 (+ 808us) server_negotiation.cc:299] Negotiation successful
0929 20:22:52.800730 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 20:22:52.802103 21588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:52.783830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:39531 (local address 127.17.53.66:56889)
0929 20:22:52.784604 (+ 774us) negotiation.cc:107] Waiting for socket to connect
0929 20:22:52.784685 (+ 81us) client_negotiation.cc:174] Beginning negotiation
0929 20:22:52.786495 (+ 1810us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:22:52.787250 (+ 755us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:22:52.787258 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:22:52.787863 (+ 605us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:22:52.788427 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:22:52.788444 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:22:52.789467 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:22:52.789472 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:22:52.790712 (+ 1240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:22:52.790725 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:22:52.791566 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:22:52.791573 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:22:52.791820 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:22:52.792533 (+ 713us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:22:52.792556 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:22:52.794804 (+ 2248us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:22:52.797797 (+ 2993us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:22:52.797806 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:22:52.797825 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:22:52.798368 (+ 543us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:22:52.798774 (+ 406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:22:52.798779 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:22:52.798782 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:22:52.799043 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:22:52.799647 (+ 604us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:22:52.799655 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:22:52.800087 (+ 432us) client_negotiation.cc:769] Sending connection context
0929 20:22:52.800378 (+ 291us) client_negotiation.cc:240] Negotiation successful
0929 20:22:52.800916 (+ 538us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":505,"thread_start_us":271,"threads_started":1}
WARNING: no policy specified for kudu/127.17.53.67@KRBTEST.COM; defaulting to no policy
I20260929 20:22:52.803777 21586 heartbeater.cc:347] Connected to a master server at 127.17.53.126:39531
I20260929 20:22:52.804077 21586 heartbeater.cc:464] Registering TS with master...
I20260929 20:22:52.804788 21586 heartbeater.cc:511] Master 127.17.53.126:39531 requested a full tablet report, sending...
I20260929 20:22:52.806308 21259 ts_manager.cc:194] Registered new tserver with Master: 1a17b9fa720d42c6a84068519de73682 (127.17.53.66:32915)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.53.67@KRBTEST.COM" created.
I20260929 20:22:52.807439 21259 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.53.66@KRBTEST.COM'} at 127.17.53.66:56889
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.53.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.67.keytab.
Entry for principal kudu/127.17.53.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.67.keytab.
I20260929 20:22:52.830765 17621 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
/tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790713339161492-17621-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790713339161492-17621-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790713339161492-17621-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790713339161492-17621-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.53.67:0
--local_ip_for_outbound_sockets=127.17.53.67
--webserver_interface=127.17.53.67
--webserver_port=0
--tserver_master_addrs=127.17.53.126:39531
--keytab_file=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.67.keytab
--principal=kudu/127.17.53.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.53.84:42555
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790713339161492-17621-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790713339161492-17621-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790713339161492-17621-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:22:53.159379 21591 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:22:53.159669 21591 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:22:53.159860 21591 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:22:53.167214 21591 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:22:53.167317 21591 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:22:53.167557 21591 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.53.67
I20260929 20:22:53.177400 21591 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.53.84:42555
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790713339161492-17621-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790713339161492-17621-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.67.keytab
--principal=kudu/127.17.53.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.53.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790713339161492-17621-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.17.53.67
--webserver_port=0
--tserver_master_addrs=127.17.53.126:39531
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.53.67
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790713339161492-17621-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:22:53.178704 21591 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:22:53.180007 21591 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:22:53.193228 21597 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:22:53.193246 21596 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:22:53.193485 21599 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:22:53.194067 21591 server_base.cc:1034] running on GCE node
I20260929 20:22:53.194599 21591 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:22:53.195603 21591 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:22:53.196817 21591 hybrid_clock.cc:648] HybridClock initialized: now 1790713373196764 us; error 39 us; skew 500 ppm
I20260929 20:22:53.197082 21591 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:22:53 dist-test-slave-zbnc krb5kdc[21213](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713373, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.53.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:22:53.200454 21591 init.cc:377] Logged in from keytab as kudu/127.17.53.67@KRBTEST.COM (short username kudu)
I20260929 20:22:53.201875 21591 webserver.cc:460] Webserver started at http://127.17.53.67:42519/ using document root <none> and password file <none>
I20260929 20:22:53.202549 21591 fs_manager.cc:359] Metadata directory not provided
I20260929 20:22:53.202677 21591 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:22:53.203048 21591 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:22:53.205894 21591 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790713339161492-17621-0/minicluster-data/ts-2/data/instance:
uuid: "2ef4327a0cf7483b8fc6f5b95765c62d"
format_stamp: "Formatted at 2026-09-29 20:22:53 on dist-test-slave-zbnc"
server_key: "b72a1683774e362a044ed47a8487e7a3"
server_key_iv: "3727048d65a61dfa17c02331e3888cbe"
server_key_version: "encryptionkey@0"
I20260929 20:22:53.206670 21591 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790713339161492-17621-0/minicluster-data/ts-2/wal/instance:
uuid: "2ef4327a0cf7483b8fc6f5b95765c62d"
format_stamp: "Formatted at 2026-09-29 20:22:53 on dist-test-slave-zbnc"
server_key: "b72a1683774e362a044ed47a8487e7a3"
server_key_iv: "3727048d65a61dfa17c02331e3888cbe"
server_key_version: "encryptionkey@0"
I20260929 20:22:53.211711 21591 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260929 20:22:53.215346 21606 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:22:53.216645 21591 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 20:22:53.216820 21591 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790713339161492-17621-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790713339161492-17621-0/minicluster-data/ts-2/wal
uuid: "2ef4327a0cf7483b8fc6f5b95765c62d"
format_stamp: "Formatted at 2026-09-29 20:22:53 on dist-test-slave-zbnc"
server_key: "b72a1683774e362a044ed47a8487e7a3"
server_key_iv: "3727048d65a61dfa17c02331e3888cbe"
server_key_version: "encryptionkey@0"
I20260929 20:22:53.217031 21591 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790713339161492-17621-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790713339161492-17621-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790713339161492-17621-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:22:53.244974 21591 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:22:53.249220 21591 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:22:53.249521 21591 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:22:53.250458 21591 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:22:53.251808 21591 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:22:53.251914 21591 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:22:53.252058 21591 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:22:53.252130 21591 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:22:53.287166 21591 rpc_server.cc:307] RPC server started. Bound to: 127.17.53.67:34487
I20260929 20:22:53.287223 21719 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.53.67:34487 every 8 connection(s)
I20260929 20:22:53.288537 21591 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790713339161492-17621-0/minicluster-data/ts-2/data/info.pb
I20260929 20:22:53.290103 17621 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu as pid 21591
I20260929 20:22:53.290354 17621 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790713339161492-17621-0/minicluster-data/ts-2/wal/instance
I20260929 20:22:53.291038 17621 external_mini_cluster.cc:1428] Setting key 9d003ca95d641c002e64fe50aeadcd89
I20260929 20:22:53.296871 21450 heartbeater.cc:503] Master 127.17.53.126:39531 was elected leader, sending a full tablet report...
Sep 29 20:22:53 dist-test-slave-zbnc krb5kdc[21213](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713373, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.53.67@KRBTEST.COM for kudu/127.17.53.126@KRBTEST.COM
I20260929 20:22:53.311345 21452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:53.291315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.53.67:45551 (local address 127.17.53.126:39531)
0929 20:22:53.291487 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0929 20:22:53.291491 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:22:53.292966 (+ 1475us) server_negotiation.cc:407] Connection header received
0929 20:22:53.296277 (+ 3311us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:22:53.296282 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:22:53.296374 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:22:53.296496 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:22:53.298467 (+ 1971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:53.299369 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:53.301146 (+ 1777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:53.301569 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:53.301706 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:22:53.305173 (+ 3467us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:22:53.305179 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:22:53.305182 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:22:53.305219 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:22:53.308262 (+ 3043us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:22:53.309017 (+ 755us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:22:53.309022 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:22:53.309026 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:22:53.309150 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:22:53.309591 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:22:53.309598 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:22:53.309601 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:22:53.309866 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:22:53.310025 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0929 20:22:53.310738 (+ 713us) server_negotiation.cc:299] Negotiation successful
0929 20:22:53.310990 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 20:22:53.312318 21722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:53.292064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:39531 (local address 127.17.53.67:45551)
0929 20:22:53.292811 (+ 747us) negotiation.cc:107] Waiting for socket to connect
0929 20:22:53.292856 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0929 20:22:53.295949 (+ 3093us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:22:53.296804 (+ 855us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:22:53.296816 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:22:53.297640 (+ 824us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:22:53.298294 (+ 654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:22:53.298321 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:22:53.299512 (+ 1191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:22:53.299517 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:22:53.300987 (+ 1470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:22:53.300999 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:22:53.301808 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:22:53.301815 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:22:53.302048 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:22:53.302765 (+ 717us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:22:53.302786 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:22:53.304952 (+ 2166us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:22:53.308420 (+ 3468us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:22:53.308432 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:22:53.308452 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:22:53.308885 (+ 433us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:22:53.309272 (+ 387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:22:53.309278 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:22:53.309281 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:22:53.309474 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:22:53.310002 (+ 528us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:22:53.310009 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:22:53.310366 (+ 357us) client_negotiation.cc:769] Sending connection context
0929 20:22:53.310646 (+ 280us) client_negotiation.cc:240] Negotiation successful
0929 20:22:53.311246 (+ 600us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":508,"thread_start_us":309,"threads_started":1}
I20260929 20:22:53.313634 21720 heartbeater.cc:347] Connected to a master server at 127.17.53.126:39531
I20260929 20:22:53.313970 21720 heartbeater.cc:464] Registering TS with master...
I20260929 20:22:53.314800 21720 heartbeater.cc:511] Master 127.17.53.126:39531 requested a full tablet report, sending...
I20260929 20:22:53.316144 21259 ts_manager.cc:194] Registered new tserver with Master: 2ef4327a0cf7483b8fc6f5b95765c62d (127.17.53.67:34487)
I20260929 20:22:53.316921 21259 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.53.67@KRBTEST.COM'} at 127.17.53.67:45551
I20260929 20:22:53.319886 17621 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 29 20:22:53 dist-test-slave-zbnc krb5kdc[21213](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713373, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 29 20:22:53 dist-test-slave-zbnc krb5kdc[21213](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713373, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.53.65@KRBTEST.COM
I20260929 20:22:53.367694 21727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:53.352752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33330 (local address 127.17.53.65:44091)
0929 20:22:53.353345 (+ 593us) server_negotiation.cc:206] Beginning negotiation
0929 20:22:53.353353 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 20:22:53.353379 (+ 26us) server_negotiation.cc:407] Connection header received
0929 20:22:53.353438 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:22:53.353444 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:22:53.353664 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:22:53.353800 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:22:53.355250 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:53.356298 (+ 1048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:53.357505 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:53.358075 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:53.358171 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:22:53.361954 (+ 3783us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:22:53.361967 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:22:53.361982 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:22:53.362019 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:22:53.364860 (+ 2841us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:22:53.365503 (+ 643us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:22:53.365509 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:22:53.365512 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:22:53.365635 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:22:53.365974 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:22:53.365977 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:22:53.365979 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:22:53.366309 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:22:53.366523 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0929 20:22:53.366851 (+ 328us) server_negotiation.cc:299] Negotiation successful
0929 20:22:53.367151 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":424,"thread_start_us":287,"threads_started":1}
Sep 29 20:22:53 dist-test-slave-zbnc krb5kdc[21213](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713373, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.53.126@KRBTEST.COM
I20260929 20:22:53.385882 21452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:53.372295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57084 (local address 127.17.53.126:39531)
0929 20:22:53.372496 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0929 20:22:53.372505 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 20:22:53.372800 (+ 295us) server_negotiation.cc:407] Connection header received
0929 20:22:53.372988 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:22:53.372992 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:22:53.373053 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:22:53.373157 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:22:53.374366 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:53.375147 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:53.376397 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:53.376796 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:53.376932 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:22:53.380845 (+ 3913us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:22:53.380849 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:22:53.380853 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:22:53.380886 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:22:53.383491 (+ 2605us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:22:53.384185 (+ 694us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:22:53.384189 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:22:53.384191 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:22:53.384312 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:22:53.384624 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:22:53.384629 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:22:53.384632 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:22:53.384941 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:22:53.385126 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0929 20:22:53.385375 (+ 249us) server_negotiation.cc:299] Negotiation successful
0929 20:22:53.385530 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
W20260929 20:22:53.386461 21259 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57084
I20260929 20:22:53.387509 17621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu pid 21320
I20260929 20:22:53.399811 21727 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:53.388342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33336 (local address 127.17.53.65:44091)
0929 20:22:53.388677 (+ 335us) server_negotiation.cc:206] Beginning negotiation
0929 20:22:53.388688 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 20:22:53.388707 (+ 19us) server_negotiation.cc:407] Connection header received
0929 20:22:53.388883 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:22:53.388888 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:22:53.389004 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:22:53.389099 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:22:53.390207 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:53.391005 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:53.392168 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:53.392548 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:53.392646 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:22:53.394795 (+ 2149us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:22:53.394799 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:22:53.394802 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:22:53.394832 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:22:53.397351 (+ 2519us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:22:53.397972 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:22:53.397976 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:22:53.397978 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:22:53.398113 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:22:53.398466 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:22:53.398469 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:22:53.398471 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:22:53.398716 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:22:53.398955 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0929 20:22:53.399297 (+ 342us) server_negotiation.cc:299] Negotiation successful
0929 20:22:53.399459 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260929 20:22:53.400413 21445 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:22:53.481535 17621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu with pid 21320
I20260929 20:22:53.497810 17621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu pid 21456
Sep 29 20:22:53 dist-test-slave-zbnc krb5kdc[21213](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713373, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.53.66@KRBTEST.COM
I20260929 20:22:53.515376 21732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:53.498692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34818 (local address 127.17.53.66:32915)
0929 20:22:53.499220 (+ 528us) server_negotiation.cc:206] Beginning negotiation
0929 20:22:53.499227 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:22:53.499251 (+ 24us) server_negotiation.cc:407] Connection header received
0929 20:22:53.499331 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:22:53.499337 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:22:53.499613 (+ 276us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:22:53.499796 (+ 183us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:22:53.501122 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:53.502120 (+ 998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:53.503521 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:53.504066 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:53.504221 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:22:53.508299 (+ 4078us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:22:53.508311 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:22:53.508326 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:22:53.508364 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:22:53.512405 (+ 4041us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:22:53.513260 (+ 855us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:22:53.513268 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:22:53.513270 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:22:53.513397 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:22:53.513815 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:22:53.513821 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:22:53.513829 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:22:53.514175 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:22:53.514465 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0929 20:22:53.514791 (+ 326us) server_negotiation.cc:299] Negotiation successful
0929 20:22:53.515084 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":392,"thread_start_us":269,"threads_started":1}
I20260929 20:22:53.516697 21581 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:22:53.598982 17621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu with pid 21456
I20260929 20:22:53.615266 17621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu pid 21591
Sep 29 20:22:53 dist-test-slave-zbnc krb5kdc[21213](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713373, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.53.67@KRBTEST.COM
I20260929 20:22:53.631899 21734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:53.616234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38202 (local address 127.17.53.67:34487)
0929 20:22:53.616856 (+ 622us) server_negotiation.cc:206] Beginning negotiation
0929 20:22:53.616863 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:22:53.616893 (+ 30us) server_negotiation.cc:407] Connection header received
0929 20:22:53.616954 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:22:53.616961 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:22:53.617175 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:22:53.617316 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:22:53.618593 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:53.619728 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:53.621104 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:53.621736 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:53.621835 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:22:53.626224 (+ 4389us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:22:53.626236 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:22:53.626250 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:22:53.626285 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:22:53.629197 (+ 2912us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:22:53.629838 (+ 641us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:22:53.629845 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:22:53.629848 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:22:53.629958 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:22:53.630268 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:22:53.630271 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:22:53.630273 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:22:53.630596 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:22:53.630831 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0929 20:22:53.631228 (+ 397us) server_negotiation.cc:299] Negotiation successful
0929 20:22:53.631548 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":474,"thread_start_us":307,"threads_started":1}
I20260929 20:22:53.633273 21715 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:22:53.723198 17621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu with pid 21591
I20260929 20:22:53.739171 17621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu pid 21226
I20260929 20:22:53.740449 21289 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:22:53.843178 17621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu with pid 21226
2026-09-29T20:22:53Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (3951 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790713339161492-17621-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 20:22:53 dist-test-slave-zbnc krb5kdc[21739](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 20:22:53 dist-test-slave-zbnc krb5kdc[21739](info): set up 2 sockets
Sep 29 20:22:53 dist-test-slave-zbnc krb5kdc[21739](info): commencing operation
krb5kdc: starting...
W20260929 20:22:54.933035 17621 mini_kdc.cc:121] Time spent starting KDC: real 1.048s user 0.001s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790713339161492-17621-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790713339161492-17621-0/krb5kdc/test-user.keytab.
Sep 29 20:22:55 dist-test-slave-zbnc krb5kdc[21739](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713375, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T20:22:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T20:22:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.17.53.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.53.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.53.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.126.keytab.
Entry for principal kudu/127.17.53.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.126.keytab.
I20260929 20:22:55.139194 17621 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
/tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790713339161492-17621-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790713339161492-17621-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790713339161492-17621-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790713339161492-17621-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.53.126:41603
--webserver_interface=127.17.53.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.126.keytab
--principal=kudu/127.17.53.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.53.84:32883
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.53.126:41603
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790713339161492-17621-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790713339161492-17621-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790713339161492-17621-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:22:55.463521 21752 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:22:55.463871 21752 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:22:55.464040 21752 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:22:55.471289 21752 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:22:55.471429 21752 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:22:55.471534 21752 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:22:55.471643 21752 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:22:55.471730 21752 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:22:55.481391 21752 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.53.84:32883
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790713339161492-17621-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790713339161492-17621-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.53.126:41603
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.126.keytab
--principal=kudu/127.17.53.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.53.126:41603
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790713339161492-17621-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.53.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790713339161492-17621-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:22:55.482690 21752 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:22:55.484021 21752 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:22:55.495759 21757 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:22:55.495936 21758 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:22:55.495901 21760 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:22:55.497246 21752 server_base.cc:1034] running on GCE node
I20260929 20:22:55.498011 21752 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:22:55.499456 21752 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:22:55.500715 21752 hybrid_clock.cc:648] HybridClock initialized: now 1790713375500645 us; error 60 us; skew 500 ppm
I20260929 20:22:55.501055 21752 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:22:55 dist-test-slave-zbnc krb5kdc[21739](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713375, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.53.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:22:55.504307 21752 init.cc:377] Logged in from keytab as kudu/127.17.53.126@KRBTEST.COM (short username kudu)
I20260929 20:22:55.505503 21752 fs_manager.cc:359] Metadata directory not provided
I20260929 20:22:55.505645 21752 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:22:55.505970 21752 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:22:55.508960 21752 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790713339161492-17621-0/minicluster-data/master-0/data/instance:
uuid: "af81c8985d6947f49b63c6531282b856"
format_stamp: "Formatted at 2026-09-29 20:22:55 on dist-test-slave-zbnc"
server_key: "8c11e5793565a782b3705235b59addc4"
server_key_iv: "56aa2678adc29b590125a064a8ba9c42"
server_key_version: "encryptionkey@0"
I20260929 20:22:55.509765 21752 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790713339161492-17621-0/minicluster-data/master-0/wal/instance:
uuid: "af81c8985d6947f49b63c6531282b856"
format_stamp: "Formatted at 2026-09-29 20:22:55 on dist-test-slave-zbnc"
server_key: "8c11e5793565a782b3705235b59addc4"
server_key_iv: "56aa2678adc29b590125a064a8ba9c42"
server_key_version: "encryptionkey@0"
I20260929 20:22:55.514958 21752 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.004s
I20260929 20:22:55.519338 21766 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:22:55.520846 21752 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 20:22:55.521005 21752 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790713339161492-17621-0/minicluster-data/master-0/data,/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790713339161492-17621-0/minicluster-data/master-0/wal
uuid: "af81c8985d6947f49b63c6531282b856"
format_stamp: "Formatted at 2026-09-29 20:22:55 on dist-test-slave-zbnc"
server_key: "8c11e5793565a782b3705235b59addc4"
server_key_iv: "56aa2678adc29b590125a064a8ba9c42"
server_key_version: "encryptionkey@0"
I20260929 20:22:55.521202 21752 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790713339161492-17621-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790713339161492-17621-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790713339161492-17621-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:22:55.536664 21752 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:22:55.541061 21752 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:22:55.541304 21752 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:22:55.559947 21752 rpc_server.cc:307] RPC server started. Bound to: 127.17.53.126:41603
I20260929 20:22:55.559966 21817 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.53.126:41603 every 8 connection(s)
I20260929 20:22:55.561208 21752 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790713339161492-17621-0/minicluster-data/master-0/data/info.pb
I20260929 20:22:55.565092 21818 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:22:55.568178 17621 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu as pid 21752
I20260929 20:22:55.568432 17621 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790713339161492-17621-0/minicluster-data/master-0/wal/instance
I20260929 20:22:55.569114 17621 external_mini_cluster.cc:1428] Setting key a63bcf531f4f8da8995a781f9fb0f7ee
I20260929 20:22:55.578320 21818 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P af81c8985d6947f49b63c6531282b856: Bootstrap starting.
I20260929 20:22:55.582257 21818 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P af81c8985d6947f49b63c6531282b856: Neither blocks nor log segments found. Creating new log.
I20260929 20:22:55.583616 21818 log.cc:824] T 00000000000000000000000000000000 P af81c8985d6947f49b63c6531282b856: Log is configured to *not* fsync() on all Append() calls
Sep 29 20:22:55 dist-test-slave-zbnc krb5kdc[21739](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713375, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.53.126@KRBTEST.COM
I20260929 20:22:55.586463 21818 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P af81c8985d6947f49b63c6531282b856: No bootstrap required, opened a new log
I20260929 20:22:55.592893 21821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:55.571335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46110 (local address 127.17.53.126:41603)
0929 20:22:55.572261 (+ 926us) server_negotiation.cc:206] Beginning negotiation
0929 20:22:55.572274 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 20:22:55.572319 (+ 45us) server_negotiation.cc:407] Connection header received
0929 20:22:55.574066 (+ 1747us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:22:55.574091 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:22:55.574533 (+ 442us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:22:55.574997 (+ 464us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:22:55.576846 (+ 1849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:55.578123 (+ 1277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:55.579908 (+ 1785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:55.580639 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:55.580781 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:22:55.585574 (+ 4793us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:22:55.585582 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:22:55.585597 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:22:55.585634 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:22:55.588757 (+ 3123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:22:55.589463 (+ 706us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:22:55.589471 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:22:55.589477 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:22:55.589636 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:22:55.590039 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:22:55.590048 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:22:55.590051 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:22:55.590609 (+ 558us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:22:55.590959 (+ 350us) server_negotiation.cc:1035] Waiting for connection context
0929 20:22:55.591445 (+ 486us) server_negotiation.cc:299] Negotiation successful
0929 20:22:55.591882 (+ 437us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":709,"thread_start_us":454,"threads_started":1}
I20260929 20:22:55.594775 21818 raft_consensus.cc:348] T 00000000000000000000000000000000 P af81c8985d6947f49b63c6531282b856 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af81c8985d6947f49b63c6531282b856" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 41603 } }
I20260929 20:22:55.595197 21818 raft_consensus.cc:374] T 00000000000000000000000000000000 P af81c8985d6947f49b63c6531282b856 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:22:55.595326 21818 raft_consensus.cc:729] T 00000000000000000000000000000000 P af81c8985d6947f49b63c6531282b856 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af81c8985d6947f49b63c6531282b856, State: Initialized, Role: FOLLOWER
I20260929 20:22:55.596005 21818 consensus_queue.cc:260] T 00000000000000000000000000000000 P af81c8985d6947f49b63c6531282b856 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af81c8985d6947f49b63c6531282b856" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 41603 } }
I20260929 20:22:55.596261 21818 raft_consensus.cc:388] T 00000000000000000000000000000000 P af81c8985d6947f49b63c6531282b856 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 20:22:55.596413 21818 raft_consensus.cc:482] T 00000000000000000000000000000000 P af81c8985d6947f49b63c6531282b856 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 20:22:55.596599 21818 raft_consensus.cc:3037] T 00000000000000000000000000000000 P af81c8985d6947f49b63c6531282b856 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:22:55.598719 21818 raft_consensus.cc:504] T 00000000000000000000000000000000 P af81c8985d6947f49b63c6531282b856 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af81c8985d6947f49b63c6531282b856" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 41603 } }
I20260929 20:22:55.599216 21818 leader_election.cc:302] T 00000000000000000000000000000000 P af81c8985d6947f49b63c6531282b856 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: af81c8985d6947f49b63c6531282b856; no voters:
I20260929 20:22:55.599862 21818 leader_election.cc:288] T 00000000000000000000000000000000 P af81c8985d6947f49b63c6531282b856 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 20:22:55.599998 21824 raft_consensus.cc:2781] T 00000000000000000000000000000000 P af81c8985d6947f49b63c6531282b856 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:22:55.600981 21824 raft_consensus.cc:686] T 00000000000000000000000000000000 P af81c8985d6947f49b63c6531282b856 [term 1 LEADER]: Becoming Leader. State: Replica: af81c8985d6947f49b63c6531282b856, State: Running, Role: LEADER
I20260929 20:22:55.601465 21824 consensus_queue.cc:237] T 00000000000000000000000000000000 P af81c8985d6947f49b63c6531282b856 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af81c8985d6947f49b63c6531282b856" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 41603 } }
I20260929 20:22:55.602206 21818 sys_catalog.cc:564] T 00000000000000000000000000000000 P af81c8985d6947f49b63c6531282b856 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:22:55.604310 21825 sys_catalog.cc:455] T 00000000000000000000000000000000 P af81c8985d6947f49b63c6531282b856 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "af81c8985d6947f49b63c6531282b856" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af81c8985d6947f49b63c6531282b856" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 41603 } } }
I20260929 20:22:55.604308 21826 sys_catalog.cc:455] T 00000000000000000000000000000000 P af81c8985d6947f49b63c6531282b856 [sys.catalog]: SysCatalogTable state changed. Reason: New leader af81c8985d6947f49b63c6531282b856. Latest consensus state: current_term: 1 leader_uuid: "af81c8985d6947f49b63c6531282b856" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af81c8985d6947f49b63c6531282b856" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 41603 } } }
I20260929 20:22:55.604707 21825 sys_catalog.cc:458] T 00000000000000000000000000000000 P af81c8985d6947f49b63c6531282b856 [sys.catalog]: This master's current role is: LEADER
I20260929 20:22:55.604813 21826 sys_catalog.cc:458] T 00000000000000000000000000000000 P af81c8985d6947f49b63c6531282b856 [sys.catalog]: This master's current role is: LEADER
I20260929 20:22:55.606642 21829 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:22:55.612350 21829 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:22:55.621145 21829 catalog_manager.cc:1347] Generated new cluster ID: 3741f22d3ccc41b1884332467e0fdbc0
I20260929 20:22:55.621279 21829 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:22:55.634347 21829 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 20:22:55.635704 21829 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:22:55.645951 21829 catalog_manager.cc:5868] T 00000000000000000000000000000000 P af81c8985d6947f49b63c6531282b856: Generated new TSK 0
I20260929 20:22:55.646792 21829 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.17.53.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.53.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.53.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.65.keytab.
Entry for principal kudu/127.17.53.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.65.keytab.
I20260929 20:22:55.698943 17621 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
/tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790713339161492-17621-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790713339161492-17621-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790713339161492-17621-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790713339161492-17621-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.53.65:0
--local_ip_for_outbound_sockets=127.17.53.65
--webserver_interface=127.17.53.65
--webserver_port=0
--tserver_master_addrs=127.17.53.126:41603
--keytab_file=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.65.keytab
--principal=kudu/127.17.53.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.53.84:32883
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790713339161492-17621-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790713339161492-17621-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790713339161492-17621-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:22:56.033632 21845 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:22:56.033977 21845 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:22:56.034158 21845 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:22:56.041709 21845 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:22:56.041862 21845 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:22:56.042049 21845 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.53.65
I20260929 20:22:56.052230 21845 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.53.84:32883
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790713339161492-17621-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790713339161492-17621-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.65.keytab
--principal=kudu/127.17.53.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.53.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790713339161492-17621-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.17.53.65
--webserver_port=0
--tserver_master_addrs=127.17.53.126:41603
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.53.65
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790713339161492-17621-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:22:56.053637 21845 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:22:56.055011 21845 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:22:56.068441 21850 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:22:56.068374 21853 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:22:56.068418 21851 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:22:56.069253 21845 server_base.cc:1034] running on GCE node
I20260929 20:22:56.069870 21845 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:22:56.070915 21845 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:22:56.072165 21845 hybrid_clock.cc:648] HybridClock initialized: now 1790713376072108 us; error 49 us; skew 500 ppm
I20260929 20:22:56.072489 21845 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:22:56 dist-test-slave-zbnc krb5kdc[21739](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713376, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.53.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:22:56.079321 21845 init.cc:377] Logged in from keytab as kudu/127.17.53.65@KRBTEST.COM (short username kudu)
I20260929 20:22:56.080499 21845 fs_manager.cc:359] Metadata directory not provided
I20260929 20:22:56.080598 21845 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:22:56.080878 21845 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:22:56.084184 21845 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790713339161492-17621-0/minicluster-data/ts-0/data/instance:
uuid: "ebb1b8c7ff1947a58f476358eb940c00"
format_stamp: "Formatted at 2026-09-29 20:22:56 on dist-test-slave-zbnc"
server_key: "fd55d573cd5abe0e894be5b65121eea0"
server_key_iv: "8610734986124fde3f6f0667e5fdeef6"
server_key_version: "encryptionkey@0"
I20260929 20:22:56.085058 21845 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790713339161492-17621-0/minicluster-data/ts-0/wal/instance:
uuid: "ebb1b8c7ff1947a58f476358eb940c00"
format_stamp: "Formatted at 2026-09-29 20:22:56 on dist-test-slave-zbnc"
server_key: "fd55d573cd5abe0e894be5b65121eea0"
server_key_iv: "8610734986124fde3f6f0667e5fdeef6"
server_key_version: "encryptionkey@0"
I20260929 20:22:56.090930 21845 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260929 20:22:56.094704 21859 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:22:56.096158 21845 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 20:22:56.096336 21845 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790713339161492-17621-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790713339161492-17621-0/minicluster-data/ts-0/wal
uuid: "ebb1b8c7ff1947a58f476358eb940c00"
format_stamp: "Formatted at 2026-09-29 20:22:56 on dist-test-slave-zbnc"
server_key: "fd55d573cd5abe0e894be5b65121eea0"
server_key_iv: "8610734986124fde3f6f0667e5fdeef6"
server_key_version: "encryptionkey@0"
I20260929 20:22:56.096566 21845 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790713339161492-17621-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790713339161492-17621-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790713339161492-17621-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:22:56.123332 21845 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:22:56.127691 21845 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:22:56.127988 21845 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:22:56.128965 21845 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:22:56.130355 21845 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:22:56.130445 21845 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:22:56.130592 21845 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:22:56.130697 21845 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:22:56.165226 21845 rpc_server.cc:307] RPC server started. Bound to: 127.17.53.65:40231
I20260929 20:22:56.165285 21972 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.53.65:40231 every 8 connection(s)
I20260929 20:22:56.166102 21845 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790713339161492-17621-0/minicluster-data/ts-0/data/info.pb
I20260929 20:22:56.170985 17621 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu as pid 21845
I20260929 20:22:56.171207 17621 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790713339161492-17621-0/minicluster-data/ts-0/wal/instance
I20260929 20:22:56.171828 17621 external_mini_cluster.cc:1428] Setting key d77fff59e7709424a361cf9c7b0bc48a
Sep 29 20:22:56 dist-test-slave-zbnc krb5kdc[21739](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713376, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.53.65@KRBTEST.COM for kudu/127.17.53.126@KRBTEST.COM
I20260929 20:22:56.187400 21975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:56.168806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.53.65:54241 (local address 127.17.53.126:41603)
0929 20:22:56.169252 (+ 446us) server_negotiation.cc:206] Beginning negotiation
0929 20:22:56.169257 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:22:56.170287 (+ 1030us) server_negotiation.cc:407] Connection header received
0929 20:22:56.172301 (+ 2014us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:22:56.172306 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:22:56.172392 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:22:56.172522 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:22:56.174103 (+ 1581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:56.174833 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:56.176459 (+ 1626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:56.176861 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:56.177006 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:22:56.180871 (+ 3865us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:22:56.180876 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:22:56.180879 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:22:56.180913 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:22:56.183967 (+ 3054us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:22:56.184954 (+ 987us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:22:56.184963 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:22:56.184967 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:22:56.185091 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:22:56.185582 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:22:56.185585 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:22:56.185592 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:22:56.185840 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:22:56.185956 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0929 20:22:56.186971 (+ 1015us) server_negotiation.cc:299] Negotiation successful
0929 20:22:56.187142 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":322,"thread_start_us":215,"threads_started":1}
I20260929 20:22:56.189122 21976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:56.169389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:41603 (local address 127.17.53.65:54241)
0929 20:22:56.170100 (+ 711us) negotiation.cc:107] Waiting for socket to connect
0929 20:22:56.170146 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0929 20:22:56.172016 (+ 1870us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:22:56.172814 (+ 798us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:22:56.172822 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:22:56.173342 (+ 520us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:22:56.173924 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:22:56.173941 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:22:56.175032 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:22:56.175042 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:22:56.176301 (+ 1259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:22:56.176312 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:22:56.177104 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:22:56.177112 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:22:56.177348 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:22:56.178048 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:22:56.178069 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:22:56.180610 (+ 2541us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:22:56.184115 (+ 3505us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:22:56.184128 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:22:56.184148 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:22:56.184822 (+ 674us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:22:56.185208 (+ 386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:22:56.185215 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:22:56.185221 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:22:56.185473 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:22:56.185979 (+ 506us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:22:56.185990 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:22:56.186493 (+ 503us) client_negotiation.cc:769] Sending connection context
0929 20:22:56.186887 (+ 394us) client_negotiation.cc:240] Negotiation successful
0929 20:22:56.187479 (+ 592us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":459,"thread_start_us":289,"threads_started":1}
I20260929 20:22:56.190913 21973 heartbeater.cc:347] Connected to a master server at 127.17.53.126:41603
I20260929 20:22:56.191220 21973 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.17.53.66@KRBTEST.COM; defaulting to no policy
I20260929 20:22:56.191884 21973 heartbeater.cc:511] Master 127.17.53.126:41603 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.53.66@KRBTEST.COM" created.
I20260929 20:22:56.194079 21783 ts_manager.cc:194] Registered new tserver with Master: ebb1b8c7ff1947a58f476358eb940c00 (127.17.53.65:40231)
I20260929 20:22:56.195602 21783 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.53.65@KRBTEST.COM'} at 127.17.53.65:54241
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.53.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.66.keytab.
Entry for principal kudu/127.17.53.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.66.keytab.
I20260929 20:22:56.218743 17621 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
/tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790713339161492-17621-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790713339161492-17621-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790713339161492-17621-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790713339161492-17621-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.53.66:0
--local_ip_for_outbound_sockets=127.17.53.66
--webserver_interface=127.17.53.66
--webserver_port=0
--tserver_master_addrs=127.17.53.126:41603
--keytab_file=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.66.keytab
--principal=kudu/127.17.53.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.53.84:32883
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790713339161492-17621-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790713339161492-17621-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790713339161492-17621-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:22:56.568352 21979 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:22:56.568655 21979 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:22:56.568846 21979 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:22:56.577180 21979 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:22:56.577297 21979 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:22:56.577486 21979 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.53.66
I20260929 20:22:56.588709 21979 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.53.84:32883
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790713339161492-17621-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790713339161492-17621-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.66.keytab
--principal=kudu/127.17.53.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.53.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790713339161492-17621-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.17.53.66
--webserver_port=0
--tserver_master_addrs=127.17.53.126:41603
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.53.66
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790713339161492-17621-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:22:56.590107 21979 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:22:56.591570 21979 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:22:56.606374 21985 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:22:56.606333 21987 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:22:56.606575 21984 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:22:56.612589 21979 server_base.cc:1034] running on GCE node
I20260929 20:22:56.613266 21979 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:22:56.614269 21979 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:22:56.615500 21979 hybrid_clock.cc:648] HybridClock initialized: now 1790713376615405 us; error 90 us; skew 500 ppm
I20260929 20:22:56.615832 21979 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:22:56 dist-test-slave-zbnc krb5kdc[21739](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713376, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.53.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:22:56.619154 21979 init.cc:377] Logged in from keytab as kudu/127.17.53.66@KRBTEST.COM (short username kudu)
I20260929 20:22:56.620642 21979 fs_manager.cc:359] Metadata directory not provided
I20260929 20:22:56.620791 21979 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:22:56.621187 21979 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:22:56.625141 21979 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790713339161492-17621-0/minicluster-data/ts-1/data/instance:
uuid: "16e5c510ebaa48bab870a608c941bf2a"
format_stamp: "Formatted at 2026-09-29 20:22:56 on dist-test-slave-zbnc"
server_key: "c5e1eed5224e46fa3909e89b0db11c96"
server_key_iv: "f0be615337d776ab6bab903fae25b10e"
server_key_version: "encryptionkey@0"
I20260929 20:22:56.626233 21979 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790713339161492-17621-0/minicluster-data/ts-1/wal/instance:
uuid: "16e5c510ebaa48bab870a608c941bf2a"
format_stamp: "Formatted at 2026-09-29 20:22:56 on dist-test-slave-zbnc"
server_key: "c5e1eed5224e46fa3909e89b0db11c96"
server_key_iv: "f0be615337d776ab6bab903fae25b10e"
server_key_version: "encryptionkey@0"
I20260929 20:22:56.631903 21979 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 20:22:56.635239 21993 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:22:56.636533 21979 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 20:22:56.636768 21979 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790713339161492-17621-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790713339161492-17621-0/minicluster-data/ts-1/wal
uuid: "16e5c510ebaa48bab870a608c941bf2a"
format_stamp: "Formatted at 2026-09-29 20:22:56 on dist-test-slave-zbnc"
server_key: "c5e1eed5224e46fa3909e89b0db11c96"
server_key_iv: "f0be615337d776ab6bab903fae25b10e"
server_key_version: "encryptionkey@0"
I20260929 20:22:56.636996 21979 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790713339161492-17621-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790713339161492-17621-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790713339161492-17621-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:22:56.666004 21979 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:22:56.670280 21979 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:22:56.670583 21979 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:22:56.671620 21979 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:22:56.673017 21979 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:22:56.673156 21979 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:22:56.673314 21979 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:22:56.673408 21979 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:22:56.707267 21979 rpc_server.cc:307] RPC server started. Bound to: 127.17.53.66:37737
I20260929 20:22:56.707324 22105 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.53.66:37737 every 8 connection(s)
I20260929 20:22:56.708160 21979 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790713339161492-17621-0/minicluster-data/ts-1/data/info.pb
I20260929 20:22:56.708550 17621 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu as pid 21979
I20260929 20:22:56.708760 17621 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790713339161492-17621-0/minicluster-data/ts-1/wal/instance
I20260929 20:22:56.709352 17621 external_mini_cluster.cc:1428] Setting key efcbc4ff08646cd01323c2b1279b36bc
Sep 29 20:22:56 dist-test-slave-zbnc krb5kdc[21739](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713376, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.53.66@KRBTEST.COM for kudu/127.17.53.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.17.53.67@KRBTEST.COM; defaulting to no policy
I20260929 20:22:56.730070 22108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:56.710775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.53.66:59887 (local address 127.17.53.126:41603)
0929 20:22:56.711232 (+ 457us) server_negotiation.cc:206] Beginning negotiation
0929 20:22:56.711238 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:22:56.712466 (+ 1228us) server_negotiation.cc:407] Connection header received
0929 20:22:56.714278 (+ 1812us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:22:56.714285 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:22:56.714375 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:22:56.714492 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:22:56.716015 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:56.716763 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:56.719389 (+ 2626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:56.719836 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:56.719930 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:22:56.724179 (+ 4249us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:22:56.724184 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:22:56.724188 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:22:56.724220 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:22:56.726975 (+ 2755us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:22:56.727758 (+ 783us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:22:56.727763 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:22:56.727767 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:22:56.727901 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:22:56.728340 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:22:56.728345 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:22:56.728353 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:22:56.728620 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:22:56.728779 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0929 20:22:56.729529 (+ 750us) server_negotiation.cc:299] Negotiation successful
0929 20:22:56.729688 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":344,"thread_start_us":239,"threads_started":1}
I20260929 20:22:56.731051 22109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:56.711477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:41603 (local address 127.17.53.66:59887)
0929 20:22:56.712281 (+ 804us) negotiation.cc:107] Waiting for socket to connect
0929 20:22:56.712329 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0929 20:22:56.714016 (+ 1687us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:22:56.714686 (+ 670us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:22:56.714694 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:22:56.715278 (+ 584us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:22:56.715827 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:22:56.715846 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:22:56.716928 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:22:56.716935 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:22:56.719255 (+ 2320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:22:56.719271 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:22:56.720072 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:22:56.720081 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:22:56.720394 (+ 313us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:22:56.721143 (+ 749us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:22:56.721176 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:22:56.723918 (+ 2742us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:22:56.727150 (+ 3232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:22:56.727160 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:22:56.727178 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:22:56.727626 (+ 448us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:22:56.728031 (+ 405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:22:56.728036 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:22:56.728039 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:22:56.728220 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:22:56.728787 (+ 567us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:22:56.728796 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:22:56.729192 (+ 396us) client_negotiation.cc:769] Sending connection context
0929 20:22:56.729439 (+ 247us) client_negotiation.cc:240] Negotiation successful
0929 20:22:56.729870 (+ 431us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":527,"thread_start_us":351,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.53.67@KRBTEST.COM" created.
I20260929 20:22:56.732623 22106 heartbeater.cc:347] Connected to a master server at 127.17.53.126:41603
I20260929 20:22:56.732941 22106 heartbeater.cc:464] Registering TS with master...
I20260929 20:22:56.733629 22106 heartbeater.cc:511] Master 127.17.53.126:41603 requested a full tablet report, sending...
I20260929 20:22:56.735095 21783 ts_manager.cc:194] Registered new tserver with Master: 16e5c510ebaa48bab870a608c941bf2a (127.17.53.66:37737)
I20260929 20:22:56.735903 21783 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.53.66@KRBTEST.COM'} at 127.17.53.66:59887
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.53.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.67.keytab.
Entry for principal kudu/127.17.53.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.67.keytab.
I20260929 20:22:56.758666 17621 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
/tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790713339161492-17621-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790713339161492-17621-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790713339161492-17621-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790713339161492-17621-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.53.67:0
--local_ip_for_outbound_sockets=127.17.53.67
--webserver_interface=127.17.53.67
--webserver_port=0
--tserver_master_addrs=127.17.53.126:41603
--keytab_file=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.67.keytab
--principal=kudu/127.17.53.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.53.84:32883
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790713339161492-17621-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790713339161492-17621-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790713339161492-17621-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:22:57.087376 22112 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:22:57.087661 22112 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:22:57.087839 22112 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:22:57.095326 22112 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:22:57.095439 22112 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:22:57.095659 22112 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.53.67
I20260929 20:22:57.106518 22112 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.53.84:32883
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790713339161492-17621-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790713339161492-17621-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.67.keytab
--principal=kudu/127.17.53.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.53.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790713339161492-17621-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.17.53.67
--webserver_port=0
--tserver_master_addrs=127.17.53.126:41603
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.53.67
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790713339161492-17621-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:22:57.107967 22112 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:22:57.109309 22112 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:22:57.122392 22120 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:22:57.122468 22118 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:22:57.122418 22117 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:22:57.123240 22112 server_base.cc:1034] running on GCE node
I20260929 20:22:57.123919 22112 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:22:57.125041 22112 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:22:57.126315 22112 hybrid_clock.cc:648] HybridClock initialized: now 1790713377126251 us; error 47 us; skew 500 ppm
I20260929 20:22:57.126706 22112 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:22:57 dist-test-slave-zbnc krb5kdc[21739](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713377, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.53.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:22:57.130015 22112 init.cc:377] Logged in from keytab as kudu/127.17.53.67@KRBTEST.COM (short username kudu)
I20260929 20:22:57.131244 22112 fs_manager.cc:359] Metadata directory not provided
I20260929 20:22:57.131336 22112 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:22:57.131634 22112 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:22:57.134559 22112 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790713339161492-17621-0/minicluster-data/ts-2/data/instance:
uuid: "c6bec5b5e56c4b94ab130be750fce9d9"
format_stamp: "Formatted at 2026-09-29 20:22:57 on dist-test-slave-zbnc"
server_key: "d347d0d674e7f37a3d9683b6ae345c57"
server_key_iv: "d464f820c13dc9d794e00b224a98e0d8"
server_key_version: "encryptionkey@0"
I20260929 20:22:57.135401 22112 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790713339161492-17621-0/minicluster-data/ts-2/wal/instance:
uuid: "c6bec5b5e56c4b94ab130be750fce9d9"
format_stamp: "Formatted at 2026-09-29 20:22:57 on dist-test-slave-zbnc"
server_key: "d347d0d674e7f37a3d9683b6ae345c57"
server_key_iv: "d464f820c13dc9d794e00b224a98e0d8"
server_key_version: "encryptionkey@0"
I20260929 20:22:57.140676 22112 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 20:22:57.144258 22126 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:22:57.145570 22112 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 20:22:57.145759 22112 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790713339161492-17621-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790713339161492-17621-0/minicluster-data/ts-2/wal
uuid: "c6bec5b5e56c4b94ab130be750fce9d9"
format_stamp: "Formatted at 2026-09-29 20:22:57 on dist-test-slave-zbnc"
server_key: "d347d0d674e7f37a3d9683b6ae345c57"
server_key_iv: "d464f820c13dc9d794e00b224a98e0d8"
server_key_version: "encryptionkey@0"
I20260929 20:22:57.145973 22112 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790713339161492-17621-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790713339161492-17621-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790713339161492-17621-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:22:57.165539 22112 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:22:57.169946 22112 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:22:57.170228 22112 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:22:57.171229 22112 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:22:57.172575 22112 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:22:57.172693 22112 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:22:57.172821 22112 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:22:57.172914 22112 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:22:57.198652 21973 heartbeater.cc:503] Master 127.17.53.126:41603 was elected leader, sending a full tablet report...
I20260929 20:22:57.206488 22112 rpc_server.cc:307] RPC server started. Bound to: 127.17.53.67:43393
I20260929 20:22:57.206589 22239 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.53.67:43393 every 8 connection(s)
I20260929 20:22:57.207489 22112 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790713339161492-17621-0/minicluster-data/ts-2/data/info.pb
I20260929 20:22:57.208277 17621 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu as pid 22112
I20260929 20:22:57.208496 17621 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790713339161492-17621-0/minicluster-data/ts-2/wal/instance
I20260929 20:22:57.209048 17621 external_mini_cluster.cc:1428] Setting key f96dfafc5ecdd95017bca99c841e767d
Sep 29 20:22:57 dist-test-slave-zbnc krb5kdc[21739](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713377, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.53.67@KRBTEST.COM for kudu/127.17.53.126@KRBTEST.COM
I20260929 20:22:57.227357 22108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:57.210175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.53.67:58527 (local address 127.17.53.126:41603)
0929 20:22:57.210387 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0929 20:22:57.210394 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:22:57.212027 (+ 1633us) server_negotiation.cc:407] Connection header received
0929 20:22:57.213906 (+ 1879us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:22:57.213909 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:22:57.213974 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:22:57.214056 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:22:57.215616 (+ 1560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:57.216359 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:57.217836 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:57.218216 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:57.218308 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:22:57.221541 (+ 3233us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:22:57.221545 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:22:57.221549 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:22:57.221580 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:22:57.224307 (+ 2727us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:22:57.225084 (+ 777us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:22:57.225101 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:22:57.225103 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:22:57.225213 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:22:57.225669 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:22:57.225674 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:22:57.225681 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:22:57.225941 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:22:57.226113 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0929 20:22:57.226785 (+ 672us) server_negotiation.cc:299] Negotiation successful
0929 20:22:57.227024 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 20:22:57.228279 22242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:57.210961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:41603 (local address 127.17.53.67:58527)
0929 20:22:57.211860 (+ 899us) negotiation.cc:107] Waiting for socket to connect
0929 20:22:57.211919 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0929 20:22:57.213652 (+ 1733us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:22:57.214272 (+ 620us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:22:57.214281 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:22:57.214796 (+ 515us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:22:57.215452 (+ 656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:22:57.215470 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:22:57.216486 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:22:57.216490 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:22:57.217709 (+ 1219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:22:57.217720 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:22:57.218397 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:22:57.218402 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:22:57.218586 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:22:57.219259 (+ 673us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:22:57.219281 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:22:57.221356 (+ 2075us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:22:57.224457 (+ 3101us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:22:57.224466 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:22:57.224483 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:22:57.224958 (+ 475us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:22:57.225325 (+ 367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:22:57.225330 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:22:57.225334 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:22:57.225531 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:22:57.226079 (+ 548us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:22:57.226087 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:22:57.226412 (+ 325us) client_negotiation.cc:769] Sending connection context
0929 20:22:57.226680 (+ 268us) client_negotiation.cc:240] Negotiation successful
0929 20:22:57.227210 (+ 530us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":595,"thread_start_us":342,"threads_started":1}
I20260929 20:22:57.229632 22240 heartbeater.cc:347] Connected to a master server at 127.17.53.126:41603
I20260929 20:22:57.229943 22240 heartbeater.cc:464] Registering TS with master...
I20260929 20:22:57.230602 22240 heartbeater.cc:511] Master 127.17.53.126:41603 requested a full tablet report, sending...
I20260929 20:22:57.232074 21783 ts_manager.cc:194] Registered new tserver with Master: c6bec5b5e56c4b94ab130be750fce9d9 (127.17.53.67:43393)
I20260929 20:22:57.232887 21783 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.53.67@KRBTEST.COM'} at 127.17.53.67:58527
I20260929 20:22:57.236857 17621 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 20:22:57.253844 22108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:57.241622 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46114 (local address 127.17.53.126:41603)
0929 20:22:57.241811 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0929 20:22:57.241816 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:22:57.242308 (+ 492us) server_negotiation.cc:407] Connection header received
0929 20:22:57.242448 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:22:57.242451 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:22:57.242514 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:22:57.242595 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:22:57.244221 (+ 1626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:57.244922 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:57.246193 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:57.246602 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:57.246691 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:22:57.248940 (+ 2249us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:22:57.248945 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:22:57.248948 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:22:57.248976 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:22:57.251534 (+ 2558us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:22:57.252135 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:22:57.252139 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:22:57.252141 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:22:57.252256 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:22:57.252566 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:22:57.252569 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:22:57.252571 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:22:57.252856 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:22:57.253039 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0929 20:22:57.253332 (+ 293us) server_negotiation.cc:299] Negotiation successful
0929 20:22:57.253476 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260929 20:22:57.257784 21783 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46114:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 20:22:57.264542 21783 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 20:22:57.297626 22253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:57.276464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47068 (local address 127.17.53.66:37737)
0929 20:22:57.289009 (+ 12545us) server_negotiation.cc:206] Beginning negotiation
0929 20:22:57.289021 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 20:22:57.289053 (+ 32us) server_negotiation.cc:407] Connection header received
0929 20:22:57.289163 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:22:57.289170 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:22:57.289412 (+ 242us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:22:57.290647 (+ 1235us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:22:57.290928 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:57.292293 (+ 1365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:57.294907 (+ 2614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:57.296337 (+ 1430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:57.296476 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:22:57.296693 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0929 20:22:57.297153 (+ 460us) server_negotiation.cc:299] Negotiation successful
0929 20:22:57.297330 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12375,"thread_start_us":12275,"threads_started":1}
I20260929 20:22:57.298190 22256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:57.277878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39234 (local address 127.17.53.65:40231)
0929 20:22:57.288702 (+ 10824us) server_negotiation.cc:206] Beginning negotiation
0929 20:22:57.288713 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 20:22:57.288750 (+ 37us) server_negotiation.cc:407] Connection header received
0929 20:22:57.288877 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:22:57.288890 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:22:57.289241 (+ 351us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:22:57.289483 (+ 242us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:22:57.290554 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:57.292025 (+ 1471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:57.295132 (+ 3107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:57.296504 (+ 1372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:57.296623 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:22:57.296827 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0929 20:22:57.297693 (+ 866us) server_negotiation.cc:299] Negotiation successful
0929 20:22:57.297867 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10599,"thread_start_us":10412,"threads_started":1}
I20260929 20:22:57.298544 22252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:57.276343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.66:37737 (local address 127.0.0.1:47068)
0929 20:22:57.277109 (+ 766us) negotiation.cc:107] Waiting for socket to connect
0929 20:22:57.277150 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 20:22:57.277333 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:22:57.289700 (+ 12367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:22:57.289709 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:22:57.289750 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:22:57.290472 (+ 722us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:22:57.290484 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:22:57.292428 (+ 1944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:22:57.292434 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:22:57.294727 (+ 2293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:22:57.294743 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:22:57.296613 (+ 1870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:22:57.296622 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:22:57.296832 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:22:57.296884 (+ 52us) client_negotiation.cc:769] Sending connection context
0929 20:22:57.297023 (+ 139us) client_negotiation.cc:240] Negotiation successful
0929 20:22:57.297216 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":529,"mutex_wait_us":4,"spinlock_wait_cycles":1280,"thread_start_us":372,"threads_started":1}
I20260929 20:22:57.298663 22257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:57.277962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49820 (local address 127.17.53.67:43393)
0929 20:22:57.289375 (+ 11413us) server_negotiation.cc:206] Beginning negotiation
0929 20:22:57.289382 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:22:57.290106 (+ 724us) server_negotiation.cc:407] Connection header received
0929 20:22:57.290429 (+ 323us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:22:57.290439 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:22:57.290646 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:22:57.290797 (+ 151us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:22:57.291566 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:57.292930 (+ 1364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:57.295367 (+ 2437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:57.296348 (+ 981us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:57.296825 (+ 477us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:22:57.297049 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0929 20:22:57.298045 (+ 996us) server_negotiation.cc:299] Negotiation successful
0929 20:22:57.298162 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11246,"thread_start_us":11395,"threads_started":1}
I20260929 20:22:57.298586 22255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:57.277521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.67:43393 (local address 127.0.0.1:49820)
0929 20:22:57.289982 (+ 12461us) negotiation.cc:107] Waiting for socket to connect
0929 20:22:57.290002 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 20:22:57.290217 (+ 215us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:22:57.291005 (+ 788us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:22:57.291012 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:22:57.291041 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:22:57.291349 (+ 308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:22:57.291359 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:22:57.293106 (+ 1747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:22:57.293112 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:22:57.295228 (+ 2116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:22:57.295237 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:22:57.297684 (+ 2447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:22:57.297690 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:22:57.297850 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:22:57.297880 (+ 30us) client_negotiation.cc:769] Sending connection context
0929 20:22:57.297966 (+ 86us) client_negotiation.cc:240] Negotiation successful
0929 20:22:57.298086 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12326,"mutex_wait_us":1,"thread_start_us":12199,"threads_started":1}
I20260929 20:22:57.298620 22254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:57.277076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.65:40231 (local address 127.0.0.1:39234)
0929 20:22:57.287837 (+ 10761us) negotiation.cc:107] Waiting for socket to connect
0929 20:22:57.287864 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0929 20:22:57.288023 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:22:57.289790 (+ 1767us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:22:57.289795 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:22:57.289827 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:22:57.290351 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:22:57.290363 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:22:57.292156 (+ 1793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:22:57.292162 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:22:57.294981 (+ 2819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:22:57.294995 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:22:57.296863 (+ 1868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:22:57.296871 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:22:57.297035 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:22:57.297068 (+ 33us) client_negotiation.cc:769] Sending connection context
0929 20:22:57.297156 (+ 88us) client_negotiation.cc:240] Negotiation successful
0929 20:22:57.297274 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10574,"mutex_wait_us":99,"thread_start_us":10396,"threads_started":1}
I20260929 20:22:57.302358 22041 tablet_service.cc:1467] Processing CreateTablet for tablet 744132d0bcfe44c4bb48e41ffca43e7e (DEFAULT_TABLE table=test-table [id=3bd0ecba5cf9495dbb1d1d41f108cd99]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:22:57.302397 21907 tablet_service.cc:1467] Processing CreateTablet for tablet 744132d0bcfe44c4bb48e41ffca43e7e (DEFAULT_TABLE table=test-table [id=3bd0ecba5cf9495dbb1d1d41f108cd99]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:22:57.303730 22041 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 744132d0bcfe44c4bb48e41ffca43e7e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:22:57.303745 21907 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 744132d0bcfe44c4bb48e41ffca43e7e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:22:57.308790 22175 tablet_service.cc:1467] Processing CreateTablet for tablet 744132d0bcfe44c4bb48e41ffca43e7e (DEFAULT_TABLE table=test-table [id=3bd0ecba5cf9495dbb1d1d41f108cd99]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:22:57.310249 22175 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 744132d0bcfe44c4bb48e41ffca43e7e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:22:57.315361 22258 tablet_bootstrap.cc:492] T 744132d0bcfe44c4bb48e41ffca43e7e P 16e5c510ebaa48bab870a608c941bf2a: Bootstrap starting.
I20260929 20:22:57.318399 22259 tablet_bootstrap.cc:492] T 744132d0bcfe44c4bb48e41ffca43e7e P ebb1b8c7ff1947a58f476358eb940c00: Bootstrap starting.
I20260929 20:22:57.318768 22258 tablet_bootstrap.cc:654] T 744132d0bcfe44c4bb48e41ffca43e7e P 16e5c510ebaa48bab870a608c941bf2a: Neither blocks nor log segments found. Creating new log.
I20260929 20:22:57.321017 22258 log.cc:824] T 744132d0bcfe44c4bb48e41ffca43e7e P 16e5c510ebaa48bab870a608c941bf2a: Log is configured to *not* fsync() on all Append() calls
I20260929 20:22:57.321954 22259 tablet_bootstrap.cc:654] T 744132d0bcfe44c4bb48e41ffca43e7e P ebb1b8c7ff1947a58f476358eb940c00: Neither blocks nor log segments found. Creating new log.
I20260929 20:22:57.323220 22259 log.cc:824] T 744132d0bcfe44c4bb48e41ffca43e7e P ebb1b8c7ff1947a58f476358eb940c00: Log is configured to *not* fsync() on all Append() calls
I20260929 20:22:57.324090 22261 tablet_bootstrap.cc:492] T 744132d0bcfe44c4bb48e41ffca43e7e P c6bec5b5e56c4b94ab130be750fce9d9: Bootstrap starting.
I20260929 20:22:57.324570 22258 tablet_bootstrap.cc:492] T 744132d0bcfe44c4bb48e41ffca43e7e P 16e5c510ebaa48bab870a608c941bf2a: No bootstrap required, opened a new log
I20260929 20:22:57.324903 22258 ts_tablet_manager.cc:1397] T 744132d0bcfe44c4bb48e41ffca43e7e P 16e5c510ebaa48bab870a608c941bf2a: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.006s
I20260929 20:22:57.327414 22261 tablet_bootstrap.cc:654] T 744132d0bcfe44c4bb48e41ffca43e7e P c6bec5b5e56c4b94ab130be750fce9d9: Neither blocks nor log segments found. Creating new log.
I20260929 20:22:57.327474 22259 tablet_bootstrap.cc:492] T 744132d0bcfe44c4bb48e41ffca43e7e P ebb1b8c7ff1947a58f476358eb940c00: No bootstrap required, opened a new log
I20260929 20:22:57.327745 22259 ts_tablet_manager.cc:1397] T 744132d0bcfe44c4bb48e41ffca43e7e P ebb1b8c7ff1947a58f476358eb940c00: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260929 20:22:57.328558 22261 log.cc:824] T 744132d0bcfe44c4bb48e41ffca43e7e P c6bec5b5e56c4b94ab130be750fce9d9: Log is configured to *not* fsync() on all Append() calls
I20260929 20:22:57.331020 22261 tablet_bootstrap.cc:492] T 744132d0bcfe44c4bb48e41ffca43e7e P c6bec5b5e56c4b94ab130be750fce9d9: No bootstrap required, opened a new log
I20260929 20:22:57.331269 22261 ts_tablet_manager.cc:1397] T 744132d0bcfe44c4bb48e41ffca43e7e P c6bec5b5e56c4b94ab130be750fce9d9: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260929 20:22:57.336220 22258 raft_consensus.cc:348] T 744132d0bcfe44c4bb48e41ffca43e7e P 16e5c510ebaa48bab870a608c941bf2a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16e5c510ebaa48bab870a608c941bf2a" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 37737 } } peers { permanent_uuid: "c6bec5b5e56c4b94ab130be750fce9d9" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 43393 } } peers { permanent_uuid: "ebb1b8c7ff1947a58f476358eb940c00" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 40231 } }
I20260929 20:22:57.336684 22258 raft_consensus.cc:374] T 744132d0bcfe44c4bb48e41ffca43e7e P 16e5c510ebaa48bab870a608c941bf2a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:22:57.336781 22258 raft_consensus.cc:729] T 744132d0bcfe44c4bb48e41ffca43e7e P 16e5c510ebaa48bab870a608c941bf2a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16e5c510ebaa48bab870a608c941bf2a, State: Initialized, Role: FOLLOWER
I20260929 20:22:57.337437 22258 consensus_queue.cc:260] T 744132d0bcfe44c4bb48e41ffca43e7e P 16e5c510ebaa48bab870a608c941bf2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16e5c510ebaa48bab870a608c941bf2a" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 37737 } } peers { permanent_uuid: "c6bec5b5e56c4b94ab130be750fce9d9" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 43393 } } peers { permanent_uuid: "ebb1b8c7ff1947a58f476358eb940c00" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 40231 } }
I20260929 20:22:57.338325 22261 raft_consensus.cc:348] T 744132d0bcfe44c4bb48e41ffca43e7e P c6bec5b5e56c4b94ab130be750fce9d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16e5c510ebaa48bab870a608c941bf2a" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 37737 } } peers { permanent_uuid: "c6bec5b5e56c4b94ab130be750fce9d9" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 43393 } } peers { permanent_uuid: "ebb1b8c7ff1947a58f476358eb940c00" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 40231 } }
I20260929 20:22:57.338668 22106 heartbeater.cc:503] Master 127.17.53.126:41603 was elected leader, sending a full tablet report...
I20260929 20:22:57.338724 22261 raft_consensus.cc:374] T 744132d0bcfe44c4bb48e41ffca43e7e P c6bec5b5e56c4b94ab130be750fce9d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:22:57.338840 22261 raft_consensus.cc:729] T 744132d0bcfe44c4bb48e41ffca43e7e P c6bec5b5e56c4b94ab130be750fce9d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6bec5b5e56c4b94ab130be750fce9d9, State: Initialized, Role: FOLLOWER
I20260929 20:22:57.339425 22261 consensus_queue.cc:260] T 744132d0bcfe44c4bb48e41ffca43e7e P c6bec5b5e56c4b94ab130be750fce9d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16e5c510ebaa48bab870a608c941bf2a" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 37737 } } peers { permanent_uuid: "c6bec5b5e56c4b94ab130be750fce9d9" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 43393 } } peers { permanent_uuid: "ebb1b8c7ff1947a58f476358eb940c00" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 40231 } }
I20260929 20:22:57.340587 22240 heartbeater.cc:503] Master 127.17.53.126:41603 was elected leader, sending a full tablet report...
I20260929 20:22:57.341113 22261 ts_tablet_manager.cc:1428] T 744132d0bcfe44c4bb48e41ffca43e7e P c6bec5b5e56c4b94ab130be750fce9d9: Time spent starting tablet: real 0.010s user 0.002s sys 0.008s
I20260929 20:22:57.338161 22259 raft_consensus.cc:348] T 744132d0bcfe44c4bb48e41ffca43e7e P ebb1b8c7ff1947a58f476358eb940c00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16e5c510ebaa48bab870a608c941bf2a" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 37737 } } peers { permanent_uuid: "c6bec5b5e56c4b94ab130be750fce9d9" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 43393 } } peers { permanent_uuid: "ebb1b8c7ff1947a58f476358eb940c00" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 40231 } }
I20260929 20:22:57.341487 22259 raft_consensus.cc:374] T 744132d0bcfe44c4bb48e41ffca43e7e P ebb1b8c7ff1947a58f476358eb940c00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:22:57.341617 22259 raft_consensus.cc:729] T 744132d0bcfe44c4bb48e41ffca43e7e P ebb1b8c7ff1947a58f476358eb940c00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ebb1b8c7ff1947a58f476358eb940c00, State: Initialized, Role: FOLLOWER
I20260929 20:22:57.342059 22258 ts_tablet_manager.cc:1428] T 744132d0bcfe44c4bb48e41ffca43e7e P 16e5c510ebaa48bab870a608c941bf2a: Time spent starting tablet: real 0.017s user 0.013s sys 0.000s
I20260929 20:22:57.342212 22259 consensus_queue.cc:260] T 744132d0bcfe44c4bb48e41ffca43e7e P ebb1b8c7ff1947a58f476358eb940c00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16e5c510ebaa48bab870a608c941bf2a" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 37737 } } peers { permanent_uuid: "c6bec5b5e56c4b94ab130be750fce9d9" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 43393 } } peers { permanent_uuid: "ebb1b8c7ff1947a58f476358eb940c00" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 40231 } }
I20260929 20:22:57.344165 22259 ts_tablet_manager.cc:1428] T 744132d0bcfe44c4bb48e41ffca43e7e P ebb1b8c7ff1947a58f476358eb940c00: Time spent starting tablet: real 0.016s user 0.009s sys 0.004s
I20260929 20:22:57.411964 22265 raft_consensus.cc:482] T 744132d0bcfe44c4bb48e41ffca43e7e P c6bec5b5e56c4b94ab130be750fce9d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 20:22:57.412298 22265 raft_consensus.cc:504] T 744132d0bcfe44c4bb48e41ffca43e7e P c6bec5b5e56c4b94ab130be750fce9d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16e5c510ebaa48bab870a608c941bf2a" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 37737 } } peers { permanent_uuid: "c6bec5b5e56c4b94ab130be750fce9d9" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 43393 } } peers { permanent_uuid: "ebb1b8c7ff1947a58f476358eb940c00" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 40231 } }
I20260929 20:22:57.414117 22265 leader_election.cc:288] T 744132d0bcfe44c4bb48e41ffca43e7e P c6bec5b5e56c4b94ab130be750fce9d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 16e5c510ebaa48bab870a608c941bf2a (127.17.53.66:37737), ebb1b8c7ff1947a58f476358eb940c00 (127.17.53.65:40231)
W20260929 20:22:57.418439 21974 tablet.cc:2369] T 744132d0bcfe44c4bb48e41ffca43e7e P ebb1b8c7ff1947a58f476358eb940c00: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 20:22:57.421720 22242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:57.414460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.66:37737 (local address 127.17.53.67:45639)
0929 20:22:57.414886 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0929 20:22:57.414905 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 20:22:57.415040 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:22:57.415469 (+ 429us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:22:57.415476 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:22:57.415496 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:22:57.415689 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:22:57.415697 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:22:57.417213 (+ 1516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:22:57.417216 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:22:57.419284 (+ 2068us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:22:57.419301 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:22:57.421024 (+ 1723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:22:57.421028 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:22:57.421164 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:22:57.421188 (+ 24us) client_negotiation.cc:769] Sending connection context
0929 20:22:57.421259 (+ 71us) client_negotiation.cc:240] Negotiation successful
0929 20:22:57.421408 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"mutex_wait_us":24}
I20260929 20:22:57.421761 22253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:57.414619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.53.67:45639 (local address 127.17.53.66:37737)
0929 20:22:57.414819 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0929 20:22:57.414825 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:22:57.415134 (+ 309us) server_negotiation.cc:407] Connection header received
0929 20:22:57.415228 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:22:57.415235 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:22:57.415328 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:22:57.415452 (+ 124us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:22:57.415843 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:57.417070 (+ 1227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:57.419487 (+ 2417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:57.420859 (+ 1372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:57.421065 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:22:57.421162 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0929 20:22:57.421332 (+ 170us) server_negotiation.cc:299] Negotiation successful
0929 20:22:57.421478 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260929 20:22:57.422262 22256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:57.414725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.53.67:44453 (local address 127.17.53.65:40231)
0929 20:22:57.415048 (+ 323us) server_negotiation.cc:206] Beginning negotiation
0929 20:22:57.415059 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 20:22:57.415487 (+ 428us) server_negotiation.cc:407] Connection header received
0929 20:22:57.415613 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:22:57.415619 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:22:57.415694 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:22:57.415829 (+ 135us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:22:57.416293 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:57.417440 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:57.419810 (+ 2370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:57.421213 (+ 1403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:57.421430 (+ 217us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:22:57.421522 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0929 20:22:57.421817 (+ 295us) server_negotiation.cc:299] Negotiation successful
0929 20:22:57.421971 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260929 20:22:57.422276 22267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:57.414642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.65:40231 (local address 127.17.53.67:44453)
0929 20:22:57.415343 (+ 701us) negotiation.cc:107] Waiting for socket to connect
0929 20:22:57.415356 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 20:22:57.415488 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:22:57.415844 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:22:57.415847 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:22:57.415867 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:22:57.416119 (+ 252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:22:57.416126 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:22:57.417606 (+ 1480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:22:57.417613 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:22:57.419655 (+ 2042us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:22:57.419665 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:22:57.421449 (+ 1784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:22:57.421454 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:22:57.421632 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:22:57.421659 (+ 27us) client_negotiation.cc:769] Sending connection context
0929 20:22:57.421759 (+ 100us) client_negotiation.cc:240] Negotiation successful
0929 20:22:57.421886 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":612,"mutex_wait_us":54,"thread_start_us":391,"threads_started":1}
I20260929 20:22:57.423110 22061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "744132d0bcfe44c4bb48e41ffca43e7e" candidate_uuid: "c6bec5b5e56c4b94ab130be750fce9d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16e5c510ebaa48bab870a608c941bf2a" is_pre_election: true
I20260929 20:22:57.423300 21927 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "744132d0bcfe44c4bb48e41ffca43e7e" candidate_uuid: "c6bec5b5e56c4b94ab130be750fce9d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ebb1b8c7ff1947a58f476358eb940c00" is_pre_election: true
I20260929 20:22:57.423637 22061 raft_consensus.cc:2454] T 744132d0bcfe44c4bb48e41ffca43e7e P 16e5c510ebaa48bab870a608c941bf2a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c6bec5b5e56c4b94ab130be750fce9d9 in term 0.
I20260929 20:22:57.423807 21927 raft_consensus.cc:2454] T 744132d0bcfe44c4bb48e41ffca43e7e P ebb1b8c7ff1947a58f476358eb940c00 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c6bec5b5e56c4b94ab130be750fce9d9 in term 0.
I20260929 20:22:57.424310 22129 leader_election.cc:302] T 744132d0bcfe44c4bb48e41ffca43e7e P c6bec5b5e56c4b94ab130be750fce9d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 16e5c510ebaa48bab870a608c941bf2a, c6bec5b5e56c4b94ab130be750fce9d9; no voters:
I20260929 20:22:57.424773 22265 raft_consensus.cc:2781] T 744132d0bcfe44c4bb48e41ffca43e7e P c6bec5b5e56c4b94ab130be750fce9d9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 20:22:57.424921 22265 raft_consensus.cc:482] T 744132d0bcfe44c4bb48e41ffca43e7e P c6bec5b5e56c4b94ab130be750fce9d9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 20:22:57.425033 22265 raft_consensus.cc:3037] T 744132d0bcfe44c4bb48e41ffca43e7e P c6bec5b5e56c4b94ab130be750fce9d9 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:22:57.427330 22265 raft_consensus.cc:504] T 744132d0bcfe44c4bb48e41ffca43e7e P c6bec5b5e56c4b94ab130be750fce9d9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16e5c510ebaa48bab870a608c941bf2a" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 37737 } } peers { permanent_uuid: "c6bec5b5e56c4b94ab130be750fce9d9" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 43393 } } peers { permanent_uuid: "ebb1b8c7ff1947a58f476358eb940c00" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 40231 } }
I20260929 20:22:57.427904 22265 leader_election.cc:288] T 744132d0bcfe44c4bb48e41ffca43e7e P c6bec5b5e56c4b94ab130be750fce9d9 [CANDIDATE]: Term 1 election: Requested vote from peers 16e5c510ebaa48bab870a608c941bf2a (127.17.53.66:37737), ebb1b8c7ff1947a58f476358eb940c00 (127.17.53.65:40231)
I20260929 20:22:57.428355 22061 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "744132d0bcfe44c4bb48e41ffca43e7e" candidate_uuid: "c6bec5b5e56c4b94ab130be750fce9d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "16e5c510ebaa48bab870a608c941bf2a"
I20260929 20:22:57.428462 21927 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "744132d0bcfe44c4bb48e41ffca43e7e" candidate_uuid: "c6bec5b5e56c4b94ab130be750fce9d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ebb1b8c7ff1947a58f476358eb940c00"
I20260929 20:22:57.428557 22061 raft_consensus.cc:3037] T 744132d0bcfe44c4bb48e41ffca43e7e P 16e5c510ebaa48bab870a608c941bf2a [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:22:57.428694 21927 raft_consensus.cc:3037] T 744132d0bcfe44c4bb48e41ffca43e7e P ebb1b8c7ff1947a58f476358eb940c00 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:22:57.430800 22061 raft_consensus.cc:2454] T 744132d0bcfe44c4bb48e41ffca43e7e P 16e5c510ebaa48bab870a608c941bf2a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6bec5b5e56c4b94ab130be750fce9d9 in term 1.
I20260929 20:22:57.430814 21927 raft_consensus.cc:2454] T 744132d0bcfe44c4bb48e41ffca43e7e P ebb1b8c7ff1947a58f476358eb940c00 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6bec5b5e56c4b94ab130be750fce9d9 in term 1.
I20260929 20:22:57.431200 22129 leader_election.cc:302] T 744132d0bcfe44c4bb48e41ffca43e7e P c6bec5b5e56c4b94ab130be750fce9d9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 16e5c510ebaa48bab870a608c941bf2a, c6bec5b5e56c4b94ab130be750fce9d9; no voters:
I20260929 20:22:57.431485 22265 raft_consensus.cc:2781] T 744132d0bcfe44c4bb48e41ffca43e7e P c6bec5b5e56c4b94ab130be750fce9d9 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:22:57.431926 22265 raft_consensus.cc:686] T 744132d0bcfe44c4bb48e41ffca43e7e P c6bec5b5e56c4b94ab130be750fce9d9 [term 1 LEADER]: Becoming Leader. State: Replica: c6bec5b5e56c4b94ab130be750fce9d9, State: Running, Role: LEADER
I20260929 20:22:57.432431 22265 consensus_queue.cc:237] T 744132d0bcfe44c4bb48e41ffca43e7e P c6bec5b5e56c4b94ab130be750fce9d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16e5c510ebaa48bab870a608c941bf2a" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 37737 } } peers { permanent_uuid: "c6bec5b5e56c4b94ab130be750fce9d9" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 43393 } } peers { permanent_uuid: "ebb1b8c7ff1947a58f476358eb940c00" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 40231 } }
I20260929 20:22:57.437119 21783 catalog_manager.cc:5495] T 744132d0bcfe44c4bb48e41ffca43e7e P c6bec5b5e56c4b94ab130be750fce9d9 reported cstate change: term changed from 0 to 1, leader changed from <none> to c6bec5b5e56c4b94ab130be750fce9d9 (127.17.53.67). New cstate: current_term: 1 leader_uuid: "c6bec5b5e56c4b94ab130be750fce9d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "16e5c510ebaa48bab870a608c941bf2a" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 37737 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c6bec5b5e56c4b94ab130be750fce9d9" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 43393 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ebb1b8c7ff1947a58f476358eb940c00" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 40231 } health_report { overall_health: UNKNOWN } } }
I20260929 20:22:57.453845 22257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:57.448310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49834 (local address 127.17.53.67:43393)
0929 20:22:57.448533 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0929 20:22:57.448539 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:22:57.448650 (+ 111us) server_negotiation.cc:407] Connection header received
0929 20:22:57.448902 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:22:57.448906 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:22:57.449001 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:22:57.449153 (+ 152us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 20:22:57.449533 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:57.450309 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:57.451717 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:57.452113 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:57.452270 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:22:57.452757 (+ 487us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 20:22:57.453277 (+ 520us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 20:22:57.453462 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0929 20:22:57.453510 (+ 48us) server_negotiation.cc:299] Negotiation successful
0929 20:22:57.453588 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
W20260929 20:22:57.459100 22241 tablet.cc:2369] T 744132d0bcfe44c4bb48e41ffca43e7e P c6bec5b5e56c4b94ab130be750fce9d9: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 20:22:57.460058 22107 tablet.cc:2369] T 744132d0bcfe44c4bb48e41ffca43e7e P 16e5c510ebaa48bab870a608c941bf2a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 20:22:57.462100 21927 raft_consensus.cc:1261] T 744132d0bcfe44c4bb48e41ffca43e7e P ebb1b8c7ff1947a58f476358eb940c00 [term 1 FOLLOWER]: Refusing update from remote peer c6bec5b5e56c4b94ab130be750fce9d9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:22:57.462127 22061 raft_consensus.cc:1261] T 744132d0bcfe44c4bb48e41ffca43e7e P 16e5c510ebaa48bab870a608c941bf2a [term 1 FOLLOWER]: Refusing update from remote peer c6bec5b5e56c4b94ab130be750fce9d9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:22:57.463125 22268 consensus_queue.cc:1035] T 744132d0bcfe44c4bb48e41ffca43e7e P c6bec5b5e56c4b94ab130be750fce9d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ebb1b8c7ff1947a58f476358eb940c00" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 40231 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:22:57.463451 22265 consensus_queue.cc:1035] T 744132d0bcfe44c4bb48e41ffca43e7e P c6bec5b5e56c4b94ab130be750fce9d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "16e5c510ebaa48bab870a608c941bf2a" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 37737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:22:57.475204 22271 mvcc.cc:205] Tried to move back new op lower bound from 7334761994074320896 to 7334761993967157248. Current Snapshot: MvccSnapshot[applied={T|T < 7334761994074320896}]
I20260929 20:22:57.489707 21783 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46114:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 20:22:57.489923 21783 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46114:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 20:22:57.493498 21783 catalog_manager.cc:5782] T 00000000000000000000000000000000 P af81c8985d6947f49b63c6531282b856: Sending DeleteTablet for 3 replicas of tablet 744132d0bcfe44c4bb48e41ffca43e7e
I20260929 20:22:57.494669 22041 tablet_service.cc:1514] Processing DeleteTablet for tablet 744132d0bcfe44c4bb48e41ffca43e7e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:22:57 UTC) from {username='kudu', principal='kudu/127.17.53.126@KRBTEST.COM'} at 127.0.0.1:47068
I20260929 20:22:57.494725 21907 tablet_service.cc:1514] Processing DeleteTablet for tablet 744132d0bcfe44c4bb48e41ffca43e7e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:22:57 UTC) from {username='kudu', principal='kudu/127.17.53.126@KRBTEST.COM'} at 127.0.0.1:39234
I20260929 20:22:57.494918 22175 tablet_service.cc:1514] Processing DeleteTablet for tablet 744132d0bcfe44c4bb48e41ffca43e7e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:22:57 UTC) from {username='kudu', principal='kudu/127.17.53.126@KRBTEST.COM'} at 127.0.0.1:49820
I20260929 20:22:57.496104 17621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu pid 21845
I20260929 20:22:57.516618 22281 tablet_replica.cc:330] T 744132d0bcfe44c4bb48e41ffca43e7e P 16e5c510ebaa48bab870a608c941bf2a: stopping tablet replica
I20260929 20:22:57.517169 22281 raft_consensus.cc:2229] T 744132d0bcfe44c4bb48e41ffca43e7e P 16e5c510ebaa48bab870a608c941bf2a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 20:22:57.517812 22281 raft_consensus.cc:2258] T 744132d0bcfe44c4bb48e41ffca43e7e P 16e5c510ebaa48bab870a608c941bf2a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:22:57.519368 22283 tablet_replica.cc:330] T 744132d0bcfe44c4bb48e41ffca43e7e P c6bec5b5e56c4b94ab130be750fce9d9: stopping tablet replica
I20260929 20:22:57.519850 22283 raft_consensus.cc:2229] T 744132d0bcfe44c4bb48e41ffca43e7e P c6bec5b5e56c4b94ab130be750fce9d9 [term 1 LEADER]: Raft consensus shutting down.
I20260929 20:22:57.520325 22282 tablet_replica.cc:330] T 744132d0bcfe44c4bb48e41ffca43e7e P ebb1b8c7ff1947a58f476358eb940c00: stopping tablet replica
I20260929 20:22:57.520612 22283 raft_consensus.cc:2258] T 744132d0bcfe44c4bb48e41ffca43e7e P c6bec5b5e56c4b94ab130be750fce9d9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:22:57.520706 22282 raft_consensus.cc:2229] T 744132d0bcfe44c4bb48e41ffca43e7e P ebb1b8c7ff1947a58f476358eb940c00 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 20:22:57.520934 22281 ts_tablet_manager.cc:1905] T 744132d0bcfe44c4bb48e41ffca43e7e P 16e5c510ebaa48bab870a608c941bf2a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:22:57.521171 22282 raft_consensus.cc:2258] T 744132d0bcfe44c4bb48e41ffca43e7e P ebb1b8c7ff1947a58f476358eb940c00 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:22:57.523159 22282 ts_tablet_manager.cc:1905] T 744132d0bcfe44c4bb48e41ffca43e7e P ebb1b8c7ff1947a58f476358eb940c00: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:22:57.526232 22283 ts_tablet_manager.cc:1905] T 744132d0bcfe44c4bb48e41ffca43e7e P c6bec5b5e56c4b94ab130be750fce9d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:22:57.529402 22281 ts_tablet_manager.cc:1918] T 744132d0bcfe44c4bb48e41ffca43e7e P 16e5c510ebaa48bab870a608c941bf2a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 20:22:57.529613 22281 log.cc:1196] T 744132d0bcfe44c4bb48e41ffca43e7e P 16e5c510ebaa48bab870a608c941bf2a: Deleting WAL directory at /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790713339161492-17621-0/minicluster-data/ts-1/wal/wals/744132d0bcfe44c4bb48e41ffca43e7e
I20260929 20:22:57.530207 22281 ts_tablet_manager.cc:1939] T 744132d0bcfe44c4bb48e41ffca43e7e P 16e5c510ebaa48bab870a608c941bf2a: Deleting consensus metadata
I20260929 20:22:57.532140 21769 catalog_manager.cc:4841] TS 16e5c510ebaa48bab870a608c941bf2a (127.17.53.66:37737): tablet 744132d0bcfe44c4bb48e41ffca43e7e (table test-table [id=3bd0ecba5cf9495dbb1d1d41f108cd99]) successfully deleted
Sep 29 20:22:57 dist-test-slave-zbnc krb5kdc[21739](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713375, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.53.65@KRBTEST.COM
I20260929 20:22:57.533468 22282 ts_tablet_manager.cc:1918] T 744132d0bcfe44c4bb48e41ffca43e7e P ebb1b8c7ff1947a58f476358eb940c00: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 20:22:57.533653 22282 log.cc:1196] T 744132d0bcfe44c4bb48e41ffca43e7e P ebb1b8c7ff1947a58f476358eb940c00: Deleting WAL directory at /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790713339161492-17621-0/minicluster-data/ts-0/wal/wals/744132d0bcfe44c4bb48e41ffca43e7e
I20260929 20:22:57.533990 22283 ts_tablet_manager.cc:1918] T 744132d0bcfe44c4bb48e41ffca43e7e P c6bec5b5e56c4b94ab130be750fce9d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 20:22:57.534164 22283 log.cc:1196] T 744132d0bcfe44c4bb48e41ffca43e7e P c6bec5b5e56c4b94ab130be750fce9d9: Deleting WAL directory at /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790713339161492-17621-0/minicluster-data/ts-2/wal/wals/744132d0bcfe44c4bb48e41ffca43e7e
I20260929 20:22:57.534195 22282 ts_tablet_manager.cc:1939] T 744132d0bcfe44c4bb48e41ffca43e7e P ebb1b8c7ff1947a58f476358eb940c00: Deleting consensus metadata
I20260929 20:22:57.534736 22283 ts_tablet_manager.cc:1939] T 744132d0bcfe44c4bb48e41ffca43e7e P c6bec5b5e56c4b94ab130be750fce9d9: Deleting consensus metadata
I20260929 20:22:57.536095 21767 catalog_manager.cc:4841] TS ebb1b8c7ff1947a58f476358eb940c00 (127.17.53.65:40231): tablet 744132d0bcfe44c4bb48e41ffca43e7e (table test-table [id=3bd0ecba5cf9495dbb1d1d41f108cd99]) successfully deleted
I20260929 20:22:57.536374 21769 catalog_manager.cc:4841] TS c6bec5b5e56c4b94ab130be750fce9d9 (127.17.53.67:43393): tablet 744132d0bcfe44c4bb48e41ffca43e7e (table test-table [id=3bd0ecba5cf9495dbb1d1d41f108cd99]) successfully deleted
I20260929 20:22:57.541340 22256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:57.497872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39244 (local address 127.17.53.65:40231)
0929 20:22:57.498110 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0929 20:22:57.498120 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 20:22:57.508860 (+ 10740us) server_negotiation.cc:407] Connection header received
0929 20:22:57.509211 (+ 351us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:22:57.509220 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:22:57.509334 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:22:57.509428 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:22:57.511794 (+ 2366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:57.513410 (+ 1616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:57.518481 (+ 5071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:57.519815 (+ 1334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:57.519953 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:22:57.535034 (+ 15081us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:22:57.535046 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:22:57.535062 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:22:57.535111 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:22:57.538308 (+ 3197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:22:57.539146 (+ 838us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:22:57.539153 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:22:57.539156 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:22:57.539302 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:22:57.539705 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:22:57.539710 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:22:57.539713 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:22:57.540125 (+ 412us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:22:57.540449 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0929 20:22:57.540745 (+ 296us) server_negotiation.cc:299] Negotiation successful
0929 20:22:57.540957 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260929 20:22:57.542071 21968 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:22:57.659803 17621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu with pid 21845
I20260929 20:22:57.677326 17621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu pid 21979
Sep 29 20:22:57 dist-test-slave-zbnc krb5kdc[21739](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713375, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.53.66@KRBTEST.COM
I20260929 20:22:57.693739 22253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:57.678200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47080 (local address 127.17.53.66:37737)
0929 20:22:57.678419 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0929 20:22:57.678426 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:22:57.678449 (+ 23us) server_negotiation.cc:407] Connection header received
0929 20:22:57.678547 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:22:57.678554 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:22:57.678648 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:22:57.678769 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:22:57.680010 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:57.681084 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:57.682357 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:57.683065 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:57.683175 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:22:57.687307 (+ 4132us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:22:57.687326 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:22:57.687346 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:22:57.687385 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:22:57.690949 (+ 3564us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:22:57.691545 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:22:57.691555 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:22:57.691558 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:22:57.691697 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:22:57.692038 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:22:57.692043 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:22:57.692046 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:22:57.692518 (+ 472us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:22:57.692838 (+ 320us) server_negotiation.cc:1035] Waiting for connection context
0929 20:22:57.693111 (+ 273us) server_negotiation.cc:299] Negotiation successful
0929 20:22:57.693342 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260929 20:22:57.694434 22101 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:22:57.784004 17621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu with pid 21979
I20260929 20:22:57.801682 17621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu pid 22112
Sep 29 20:22:57 dist-test-slave-zbnc krb5kdc[21739](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713375, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.53.67@KRBTEST.COM
I20260929 20:22:57.818987 22257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:57.802519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49838 (local address 127.17.53.67:43393)
0929 20:22:57.802735 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0929 20:22:57.802740 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:22:57.802789 (+ 49us) server_negotiation.cc:407] Connection header received
0929 20:22:57.803035 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:22:57.803044 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:22:57.803110 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:22:57.803193 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:22:57.804995 (+ 1802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:57.805816 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:57.807710 (+ 1894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:22:57.808245 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:22:57.808379 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:22:57.812858 (+ 4479us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:22:57.812869 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:22:57.812885 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:22:57.812920 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:22:57.816060 (+ 3140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:22:57.816790 (+ 730us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:22:57.816797 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:22:57.816800 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:22:57.816941 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:22:57.817343 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:22:57.817352 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:22:57.817355 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:22:57.817720 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:22:57.818047 (+ 327us) server_negotiation.cc:1035] Waiting for connection context
0929 20:22:57.818338 (+ 291us) server_negotiation.cc:299] Negotiation successful
0929 20:22:57.818554 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260929 20:22:57.819732 22235 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:22:57.940918 17621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu with pid 22112
I20260929 20:22:57.956418 17621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu pid 21752
I20260929 20:22:57.957598 21813 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:22:58.066135 17621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu with pid 21752
2026-09-29T20:22:58Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (4227 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-29T20:22:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T20:22:58Z Disabled control of system clock
I20260929 20:22:58.137538 17621 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
/tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790713339161492-17621-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790713339161492-17621-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790713339161492-17621-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790713339161492-17621-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.53.126:34461
--webserver_interface=127.17.53.126
--webserver_port=0
--builtin_ntp_servers=127.17.53.84:43051
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.53.126:34461
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260929 20:22:58.484131 22292 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:22:58.484454 22292 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:22:58.484557 22292 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:22:58.492313 22292 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:22:58.492432 22292 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:22:58.492479 22292 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:22:58.492590 22292 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:22:58.492686 22292 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:22:58.502208 22292 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.53.84:43051
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790713339161492-17621-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790713339161492-17621-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.53.126:34461
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.53.126:34461
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790713339161492-17621-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.53.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790713339161492-17621-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:22:58.503882 22292 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:22:58.505820 22292 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:22:58.517802 22297 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:22:58.517823 22298 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:22:58.518018 22300 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:22:58.518822 22292 server_base.cc:1034] running on GCE node
I20260929 20:22:58.519620 22292 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:22:58.521034 22292 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:22:58.522240 22292 hybrid_clock.cc:648] HybridClock initialized: now 1790713378522189 us; error 42 us; skew 500 ppm
I20260929 20:22:58.522603 22292 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 20:22:58.525156 22292 webserver.cc:460] Webserver started at http://127.17.53.126:36887/ using document root <none> and password file <none>
I20260929 20:22:58.525938 22292 fs_manager.cc:359] Metadata directory not provided
I20260929 20:22:58.526055 22292 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:22:58.526405 22292 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:22:58.530297 22292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790713339161492-17621-0/minicluster-data/master-0/data/instance:
uuid: "7a41f7c7baf64f50a72a85c232bc81c0"
format_stamp: "Formatted at 2026-09-29 20:22:58 on dist-test-slave-zbnc"
server_key: "cc7efda6ba938fa12be8218dfd3423cf"
server_key_iv: "de386b66439f6c2cb935ac2f1e98c85e"
server_key_version: "encryptionkey@0"
I20260929 20:22:58.531322 22292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790713339161492-17621-0/minicluster-data/master-0/wal/instance:
uuid: "7a41f7c7baf64f50a72a85c232bc81c0"
format_stamp: "Formatted at 2026-09-29 20:22:58 on dist-test-slave-zbnc"
server_key: "cc7efda6ba938fa12be8218dfd3423cf"
server_key_iv: "de386b66439f6c2cb935ac2f1e98c85e"
server_key_version: "encryptionkey@0"
I20260929 20:22:58.536577 22292 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260929 20:22:58.540046 22306 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:22:58.541632 22292 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260929 20:22:58.541844 22292 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790713339161492-17621-0/minicluster-data/master-0/data,/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790713339161492-17621-0/minicluster-data/master-0/wal
uuid: "7a41f7c7baf64f50a72a85c232bc81c0"
format_stamp: "Formatted at 2026-09-29 20:22:58 on dist-test-slave-zbnc"
server_key: "cc7efda6ba938fa12be8218dfd3423cf"
server_key_iv: "de386b66439f6c2cb935ac2f1e98c85e"
server_key_version: "encryptionkey@0"
I20260929 20:22:58.542122 22292 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790713339161492-17621-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790713339161492-17621-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790713339161492-17621-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:22:58.548324 22292 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:22:58.549101 22292 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:22:58.549402 22292 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:22:58.568542 22292 rpc_server.cc:307] RPC server started. Bound to: 127.17.53.126:34461
I20260929 20:22:58.568562 22357 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.53.126:34461 every 8 connection(s)
I20260929 20:22:58.569983 22292 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790713339161492-17621-0/minicluster-data/master-0/data/info.pb
I20260929 20:22:58.573896 22358 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:22:58.577133 17621 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu as pid 22292
I20260929 20:22:58.577324 17621 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790713339161492-17621-0/minicluster-data/master-0/wal/instance
I20260929 20:22:58.578014 17621 external_mini_cluster.cc:1428] Setting key e654d78c90b9a58b01c20ba7d71e09e5
I20260929 20:22:58.587631 22361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:58.580273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40490 (local address 127.17.53.126:34461)
0929 20:22:58.581416 (+ 1143us) server_negotiation.cc:206] Beginning negotiation
0929 20:22:58.581431 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 20:22:58.581481 (+ 50us) server_negotiation.cc:407] Connection header received
0929 20:22:58.583433 (+ 1952us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:22:58.583487 (+ 54us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:22:58.584084 (+ 597us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:22:58.584568 (+ 484us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:22:58.585622 (+ 1054us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:22:58.585631 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:22:58.585652 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:22:58.585670 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:22:58.585742 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:22:58.585765 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:22:58.585991 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0929 20:22:58.586123 (+ 132us) server_negotiation.cc:299] Negotiation successful
0929 20:22:58.586388 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":593,"thread_start_us":342,"threads_started":1}
I20260929 20:22:58.588168 22358 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a41f7c7baf64f50a72a85c232bc81c0: Bootstrap starting.
I20260929 20:22:58.591391 22358 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7a41f7c7baf64f50a72a85c232bc81c0: Neither blocks nor log segments found. Creating new log.
I20260929 20:22:58.592442 22358 log.cc:824] T 00000000000000000000000000000000 P 7a41f7c7baf64f50a72a85c232bc81c0: Log is configured to *not* fsync() on all Append() calls
I20260929 20:22:58.595587 22358 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a41f7c7baf64f50a72a85c232bc81c0: No bootstrap required, opened a new log
I20260929 20:22:58.602811 22358 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7a41f7c7baf64f50a72a85c232bc81c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a41f7c7baf64f50a72a85c232bc81c0" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 34461 } }
I20260929 20:22:58.603266 22358 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7a41f7c7baf64f50a72a85c232bc81c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:22:58.603382 22358 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7a41f7c7baf64f50a72a85c232bc81c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a41f7c7baf64f50a72a85c232bc81c0, State: Initialized, Role: FOLLOWER
I20260929 20:22:58.604018 22358 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7a41f7c7baf64f50a72a85c232bc81c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a41f7c7baf64f50a72a85c232bc81c0" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 34461 } }
I20260929 20:22:58.604239 22358 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7a41f7c7baf64f50a72a85c232bc81c0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 20:22:58.604367 22358 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7a41f7c7baf64f50a72a85c232bc81c0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 20:22:58.604573 22358 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7a41f7c7baf64f50a72a85c232bc81c0 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:22:58.606691 22358 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7a41f7c7baf64f50a72a85c232bc81c0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a41f7c7baf64f50a72a85c232bc81c0" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 34461 } }
I20260929 20:22:58.607316 22358 leader_election.cc:302] T 00000000000000000000000000000000 P 7a41f7c7baf64f50a72a85c232bc81c0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7a41f7c7baf64f50a72a85c232bc81c0; no voters:
I20260929 20:22:58.608100 22358 leader_election.cc:288] T 00000000000000000000000000000000 P 7a41f7c7baf64f50a72a85c232bc81c0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 20:22:58.608265 22363 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7a41f7c7baf64f50a72a85c232bc81c0 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:22:58.609087 22363 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7a41f7c7baf64f50a72a85c232bc81c0 [term 1 LEADER]: Becoming Leader. State: Replica: 7a41f7c7baf64f50a72a85c232bc81c0, State: Running, Role: LEADER
I20260929 20:22:58.609701 22363 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7a41f7c7baf64f50a72a85c232bc81c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a41f7c7baf64f50a72a85c232bc81c0" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 34461 } }
I20260929 20:22:58.610356 22358 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7a41f7c7baf64f50a72a85c232bc81c0 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:22:58.612807 22364 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a41f7c7baf64f50a72a85c232bc81c0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7a41f7c7baf64f50a72a85c232bc81c0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a41f7c7baf64f50a72a85c232bc81c0" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 34461 } } }
I20260929 20:22:58.613260 22364 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a41f7c7baf64f50a72a85c232bc81c0 [sys.catalog]: This master's current role is: LEADER
I20260929 20:22:58.615051 22365 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a41f7c7baf64f50a72a85c232bc81c0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7a41f7c7baf64f50a72a85c232bc81c0. Latest consensus state: current_term: 1 leader_uuid: "7a41f7c7baf64f50a72a85c232bc81c0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a41f7c7baf64f50a72a85c232bc81c0" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 34461 } } }
I20260929 20:22:58.615908 22365 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a41f7c7baf64f50a72a85c232bc81c0 [sys.catalog]: This master's current role is: LEADER
I20260929 20:22:58.620966 22372 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:22:58.625403 22372 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:22:58.634567 22372 catalog_manager.cc:1347] Generated new cluster ID: b731ed21ecbd4b76b79855f37481dcf1
I20260929 20:22:58.634752 22372 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:22:58.644424 22372 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 20:22:58.645005 22372 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:22:58.654773 22372 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7a41f7c7baf64f50a72a85c232bc81c0: Generated new TSK 0
I20260929 20:22:58.655671 22372 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 20:22:58.667129 17621 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
/tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790713339161492-17621-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790713339161492-17621-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790713339161492-17621-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790713339161492-17621-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.53.65:0
--local_ip_for_outbound_sockets=127.17.53.65
--webserver_interface=127.17.53.65
--webserver_port=0
--tserver_master_addrs=127.17.53.126:34461
--builtin_ntp_servers=127.17.53.84:43051
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260929 20:22:58.998582 22382 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:22:58.998885 22382 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:22:58.999023 22382 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:22:59.006353 22382 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:22:59.006464 22382 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:22:59.006646 22382 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.53.65
I20260929 20:22:59.016543 22382 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.53.84:43051
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790713339161492-17621-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790713339161492-17621-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.53.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790713339161492-17621-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.17.53.65
--webserver_port=0
--tserver_master_addrs=127.17.53.126:34461
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.53.65
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790713339161492-17621-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:22:59.017920 22382 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:22:59.019774 22382 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:22:59.033066 22390 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:22:59.033144 22388 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:22:59.033442 22387 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:22:59.033895 22382 server_base.cc:1034] running on GCE node
I20260929 20:22:59.034495 22382 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:22:59.035614 22382 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:22:59.036820 22382 hybrid_clock.cc:648] HybridClock initialized: now 1790713379036773 us; error 27 us; skew 500 ppm
I20260929 20:22:59.037174 22382 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 20:22:59.039574 22382 webserver.cc:460] Webserver started at http://127.17.53.65:37181/ using document root <none> and password file <none>
I20260929 20:22:59.040278 22382 fs_manager.cc:359] Metadata directory not provided
I20260929 20:22:59.040400 22382 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:22:59.040737 22382 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:22:59.043614 22382 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790713339161492-17621-0/minicluster-data/ts-0/data/instance:
uuid: "2fff9d527a4e458999850bbb5c1f1412"
format_stamp: "Formatted at 2026-09-29 20:22:59 on dist-test-slave-zbnc"
server_key: "85bacdcccf7f7e0897d73a7f6ddb3fdf"
server_key_iv: "588cd50d9b0cb656302bbdd201b0871e"
server_key_version: "encryptionkey@0"
I20260929 20:22:59.044404 22382 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790713339161492-17621-0/minicluster-data/ts-0/wal/instance:
uuid: "2fff9d527a4e458999850bbb5c1f1412"
format_stamp: "Formatted at 2026-09-29 20:22:59 on dist-test-slave-zbnc"
server_key: "85bacdcccf7f7e0897d73a7f6ddb3fdf"
server_key_iv: "588cd50d9b0cb656302bbdd201b0871e"
server_key_version: "encryptionkey@0"
I20260929 20:22:59.049669 22382 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 20:22:59.055433 22396 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:22:59.057097 22382 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.004s sys 0.000s
I20260929 20:22:59.057298 22382 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790713339161492-17621-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790713339161492-17621-0/minicluster-data/ts-0/wal
uuid: "2fff9d527a4e458999850bbb5c1f1412"
format_stamp: "Formatted at 2026-09-29 20:22:59 on dist-test-slave-zbnc"
server_key: "85bacdcccf7f7e0897d73a7f6ddb3fdf"
server_key_iv: "588cd50d9b0cb656302bbdd201b0871e"
server_key_version: "encryptionkey@0"
I20260929 20:22:59.057541 22382 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790713339161492-17621-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790713339161492-17621-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790713339161492-17621-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:22:59.064010 22382 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:22:59.064810 22382 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:22:59.065081 22382 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:22:59.066006 22382 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:22:59.067499 22382 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:22:59.067618 22382 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:22:59.067745 22382 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:22:59.067843 22382 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:22:59.099998 22382 rpc_server.cc:307] RPC server started. Bound to: 127.17.53.65:45127
I20260929 20:22:59.100054 22508 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.53.65:45127 every 8 connection(s)
I20260929 20:22:59.101411 22382 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790713339161492-17621-0/minicluster-data/ts-0/data/info.pb
I20260929 20:22:59.106573 17621 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu as pid 22382
I20260929 20:22:59.106812 17621 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790713339161492-17621-0/minicluster-data/ts-0/wal/instance
I20260929 20:22:59.107471 17621 external_mini_cluster.cc:1428] Setting key af90e7e6e5555422bdfd105547f115f5
I20260929 20:22:59.110284 22511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:59.104077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.53.65:53649 (local address 127.17.53.126:34461)
0929 20:22:59.104556 (+ 479us) server_negotiation.cc:206] Beginning negotiation
0929 20:22:59.104561 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:22:59.105775 (+ 1214us) server_negotiation.cc:407] Connection header received
0929 20:22:59.107778 (+ 2003us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:22:59.107784 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:22:59.107854 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:22:59.107988 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:22:59.109526 (+ 1538us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:22:59.109530 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:22:59.109533 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:22:59.109570 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:22:59.109626 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:22:59.109643 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:22:59.109767 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0929 20:22:59.109948 (+ 181us) server_negotiation.cc:299] Negotiation successful
0929 20:22:59.110006 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":363,"thread_start_us":264,"threads_started":1}
I20260929 20:22:59.110677 17621 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
/tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790713339161492-17621-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790713339161492-17621-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790713339161492-17621-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790713339161492-17621-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.53.66:0
--local_ip_for_outbound_sockets=127.17.53.66
--webserver_interface=127.17.53.66
--webserver_port=0
--tserver_master_addrs=127.17.53.126:34461
--builtin_ntp_servers=127.17.53.84:43051
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260929 20:22:59.111346 22512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:59.104601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:34461 (local address 127.17.53.65:53649)
0929 20:22:59.105594 (+ 993us) negotiation.cc:107] Waiting for socket to connect
0929 20:22:59.105645 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0929 20:22:59.107386 (+ 1741us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:22:59.108288 (+ 902us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:22:59.108303 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:22:59.108431 (+ 128us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:22:59.109154 (+ 723us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 20:22:59.109158 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:22:59.109208 (+ 50us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 20:22:59.109352 (+ 144us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:22:59.109745 (+ 393us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:22:59.109753 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:22:59.109759 (+ 6us) client_negotiation.cc:769] Sending connection context
0929 20:22:59.109978 (+ 219us) client_negotiation.cc:240] Negotiation successful
0929 20:22:59.110195 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":519,"thread_start_us":288,"threads_started":1}
I20260929 20:22:59.112883 22509 heartbeater.cc:347] Connected to a master server at 127.17.53.126:34461
I20260929 20:22:59.113224 22509 heartbeater.cc:464] Registering TS with master...
I20260929 20:22:59.113845 22509 heartbeater.cc:511] Master 127.17.53.126:34461 requested a full tablet report, sending...
I20260929 20:22:59.116094 22323 ts_manager.cc:194] Registered new tserver with Master: 2fff9d527a4e458999850bbb5c1f1412 (127.17.53.65:45127)
W20260929 20:22:59.438498 22513 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:22:59.438844 22513 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:22:59.439043 22513 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:22:59.446331 22513 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:22:59.446486 22513 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:22:59.446682 22513 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.53.66
I20260929 20:22:59.455977 22513 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.53.84:43051
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790713339161492-17621-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790713339161492-17621-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.53.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790713339161492-17621-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.17.53.66
--webserver_port=0
--tserver_master_addrs=127.17.53.126:34461
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.53.66
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790713339161492-17621-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:22:59.457284 22513 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:22:59.459182 22513 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:22:59.472242 22521 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:22:59.472362 22519 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:22:59.472373 22518 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:22:59.473435 22513 server_base.cc:1034] running on GCE node
I20260929 20:22:59.474098 22513 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:22:59.475207 22513 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:22:59.476455 22513 hybrid_clock.cc:648] HybridClock initialized: now 1790713379476365 us; error 81 us; skew 500 ppm
I20260929 20:22:59.476774 22513 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 20:22:59.479012 22513 webserver.cc:460] Webserver started at http://127.17.53.66:45513/ using document root <none> and password file <none>
I20260929 20:22:59.479745 22513 fs_manager.cc:359] Metadata directory not provided
I20260929 20:22:59.479882 22513 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:22:59.480206 22513 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:22:59.483062 22513 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790713339161492-17621-0/minicluster-data/ts-1/data/instance:
uuid: "c6eda6e706d1440093d98981edeeee80"
format_stamp: "Formatted at 2026-09-29 20:22:59 on dist-test-slave-zbnc"
server_key: "ad488882fef1c4840b47f042bf984a6b"
server_key_iv: "434e7624ac5e144c6c674265629243f4"
server_key_version: "encryptionkey@0"
I20260929 20:22:59.483857 22513 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790713339161492-17621-0/minicluster-data/ts-1/wal/instance:
uuid: "c6eda6e706d1440093d98981edeeee80"
format_stamp: "Formatted at 2026-09-29 20:22:59 on dist-test-slave-zbnc"
server_key: "ad488882fef1c4840b47f042bf984a6b"
server_key_iv: "434e7624ac5e144c6c674265629243f4"
server_key_version: "encryptionkey@0"
I20260929 20:22:59.488973 22513 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 20:22:59.492450 22527 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:22:59.493844 22513 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 20:22:59.494023 22513 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790713339161492-17621-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790713339161492-17621-0/minicluster-data/ts-1/wal
uuid: "c6eda6e706d1440093d98981edeeee80"
format_stamp: "Formatted at 2026-09-29 20:22:59 on dist-test-slave-zbnc"
server_key: "ad488882fef1c4840b47f042bf984a6b"
server_key_iv: "434e7624ac5e144c6c674265629243f4"
server_key_version: "encryptionkey@0"
I20260929 20:22:59.494251 22513 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790713339161492-17621-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790713339161492-17621-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790713339161492-17621-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:22:59.500181 22513 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:22:59.501000 22513 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:22:59.501278 22513 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:22:59.502183 22513 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:22:59.503675 22513 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:22:59.503809 22513 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:22:59.503944 22513 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:22:59.504037 22513 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:22:59.538352 22513 rpc_server.cc:307] RPC server started. Bound to: 127.17.53.66:39471
I20260929 20:22:59.538403 22639 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.53.66:39471 every 8 connection(s)
I20260929 20:22:59.539749 22513 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790713339161492-17621-0/minicluster-data/ts-1/data/info.pb
I20260929 20:22:59.541222 17621 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu as pid 22513
I20260929 20:22:59.541396 17621 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790713339161492-17621-0/minicluster-data/ts-1/wal/instance
I20260929 20:22:59.541839 17621 external_mini_cluster.cc:1428] Setting key 8762a2a8d4dbeeae216dda6895b26041
I20260929 20:22:59.544902 17621 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
/tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790713339161492-17621-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790713339161492-17621-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790713339161492-17621-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790713339161492-17621-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.53.67:0
--local_ip_for_outbound_sockets=127.17.53.67
--webserver_interface=127.17.53.67
--webserver_port=0
--tserver_master_addrs=127.17.53.126:34461
--builtin_ntp_servers=127.17.53.84:43051
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260929 20:22:59.548839 22511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:59.542509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.53.66:45785 (local address 127.17.53.126:34461)
0929 20:22:59.542722 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0929 20:22:59.542726 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:22:59.544436 (+ 1710us) server_negotiation.cc:407] Connection header received
0929 20:22:59.546349 (+ 1913us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:22:59.546354 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:22:59.546433 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:22:59.546526 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:22:59.548089 (+ 1563us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:22:59.548095 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:22:59.548099 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:22:59.548117 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:22:59.548157 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:22:59.548169 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:22:59.548291 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0929 20:22:59.548482 (+ 191us) server_negotiation.cc:299] Negotiation successful
0929 20:22:59.548549 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260929 20:22:59.549777 22642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:59.543240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:34461 (local address 127.17.53.66:45785)
0929 20:22:59.544279 (+ 1039us) negotiation.cc:107] Waiting for socket to connect
0929 20:22:59.544321 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 20:22:59.546036 (+ 1715us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:22:59.546803 (+ 767us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:22:59.546814 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:22:59.546960 (+ 146us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:22:59.547782 (+ 822us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 20:22:59.547786 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:22:59.547839 (+ 53us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 20:22:59.547925 (+ 86us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:22:59.548261 (+ 336us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:22:59.548272 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:22:59.548279 (+ 7us) client_negotiation.cc:769] Sending connection context
0929 20:22:59.548568 (+ 289us) client_negotiation.cc:240] Negotiation successful
0929 20:22:59.548832 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":545,"thread_start_us":331,"threads_started":1}
I20260929 20:22:59.551259 22640 heartbeater.cc:347] Connected to a master server at 127.17.53.126:34461
I20260929 20:22:59.551573 22640 heartbeater.cc:464] Registering TS with master...
I20260929 20:22:59.552255 22640 heartbeater.cc:511] Master 127.17.53.126:34461 requested a full tablet report, sending...
I20260929 20:22:59.553893 22323 ts_manager.cc:194] Registered new tserver with Master: c6eda6e706d1440093d98981edeeee80 (127.17.53.66:39471)
W20260929 20:22:59.879899 22643 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:22:59.880182 22643 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:22:59.880359 22643 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:22:59.887842 22643 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:22:59.887956 22643 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:22:59.888136 22643 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.53.67
I20260929 20:22:59.897614 22643 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.53.84:43051
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790713339161492-17621-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790713339161492-17621-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.53.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790713339161492-17621-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.17.53.67
--webserver_port=0
--tserver_master_addrs=127.17.53.126:34461
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.53.67
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790713339161492-17621-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:22:59.899052 22643 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:22:59.900911 22643 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:22:59.914356 22648 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:22:59.914438 22649 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:22:59.914413 22651 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:22:59.914923 22643 server_base.cc:1034] running on GCE node
I20260929 20:22:59.915853 22643 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:22:59.916755 22643 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:22:59.917992 22643 hybrid_clock.cc:648] HybridClock initialized: now 1790713379917957 us; error 48 us; skew 500 ppm
I20260929 20:22:59.918318 22643 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 20:22:59.920634 22643 webserver.cc:460] Webserver started at http://127.17.53.67:35703/ using document root <none> and password file <none>
I20260929 20:22:59.921335 22643 fs_manager.cc:359] Metadata directory not provided
I20260929 20:22:59.921476 22643 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:22:59.921824 22643 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:22:59.924875 22643 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790713339161492-17621-0/minicluster-data/ts-2/data/instance:
uuid: "0739d064d48a4eab8330d8df4a962f77"
format_stamp: "Formatted at 2026-09-29 20:22:59 on dist-test-slave-zbnc"
server_key: "91cd15942b5c5b7878b8edd832c0f2a8"
server_key_iv: "ee9e2ce5e2cf2a1fc70001cde3dba46a"
server_key_version: "encryptionkey@0"
I20260929 20:22:59.925751 22643 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790713339161492-17621-0/minicluster-data/ts-2/wal/instance:
uuid: "0739d064d48a4eab8330d8df4a962f77"
format_stamp: "Formatted at 2026-09-29 20:22:59 on dist-test-slave-zbnc"
server_key: "91cd15942b5c5b7878b8edd832c0f2a8"
server_key_iv: "ee9e2ce5e2cf2a1fc70001cde3dba46a"
server_key_version: "encryptionkey@0"
I20260929 20:22:59.931061 22643 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260929 20:22:59.934804 22657 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:22:59.936254 22643 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 20:22:59.936446 22643 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790713339161492-17621-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790713339161492-17621-0/minicluster-data/ts-2/wal
uuid: "0739d064d48a4eab8330d8df4a962f77"
format_stamp: "Formatted at 2026-09-29 20:22:59 on dist-test-slave-zbnc"
server_key: "91cd15942b5c5b7878b8edd832c0f2a8"
server_key_iv: "ee9e2ce5e2cf2a1fc70001cde3dba46a"
server_key_version: "encryptionkey@0"
I20260929 20:22:59.936672 22643 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790713339161492-17621-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790713339161492-17621-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790713339161492-17621-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:22:59.942521 22643 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:22:59.943410 22643 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:22:59.943687 22643 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:22:59.944581 22643 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:22:59.946237 22643 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:22:59.946365 22643 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:22:59.946499 22643 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:22:59.946587 22643 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:22:59.980739 22643 rpc_server.cc:307] RPC server started. Bound to: 127.17.53.67:33137
I20260929 20:22:59.980794 22769 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.53.67:33137 every 8 connection(s)
I20260929 20:22:59.982095 22643 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790713339161492-17621-0/minicluster-data/ts-2/data/info.pb
I20260929 20:22:59.986367 17621 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu as pid 22643
I20260929 20:22:59.986534 17621 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790713339161492-17621-0/minicluster-data/ts-2/wal/instance
I20260929 20:22:59.987141 17621 external_mini_cluster.cc:1428] Setting key bbe73fbe017671525292c7f218ead882
I20260929 20:22:59.990895 22511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:59.984718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.53.67:58311 (local address 127.17.53.126:34461)
0929 20:22:59.984890 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0929 20:22:59.984895 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:22:59.986378 (+ 1483us) server_negotiation.cc:407] Connection header received
0929 20:22:59.988541 (+ 2163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:22:59.988546 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:22:59.988622 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:22:59.988738 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:22:59.990141 (+ 1403us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:22:59.990147 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:22:59.990152 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:22:59.990171 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:22:59.990206 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:22:59.990218 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:22:59.990342 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0929 20:22:59.990530 (+ 188us) server_negotiation.cc:299] Negotiation successful
0929 20:22:59.990604 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 20:22:59.991902 22772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:22:59.985182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:34461 (local address 127.17.53.67:58311)
0929 20:22:59.986184 (+ 1002us) negotiation.cc:107] Waiting for socket to connect
0929 20:22:59.986235 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0929 20:22:59.988246 (+ 2011us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:22:59.988972 (+ 726us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:22:59.988981 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:22:59.989074 (+ 93us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:22:59.989783 (+ 709us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 20:22:59.989786 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:22:59.989834 (+ 48us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 20:22:59.989972 (+ 138us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:22:59.990312 (+ 340us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:22:59.990324 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:22:59.990332 (+ 8us) client_negotiation.cc:769] Sending connection context
0929 20:22:59.990613 (+ 281us) client_negotiation.cc:240] Negotiation successful
0929 20:22:59.990893 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":429,"thread_start_us":248,"threads_started":1}
I20260929 20:22:59.993175 22770 heartbeater.cc:347] Connected to a master server at 127.17.53.126:34461
I20260929 20:22:59.993443 22770 heartbeater.cc:464] Registering TS with master...
I20260929 20:22:59.993978 22770 heartbeater.cc:511] Master 127.17.53.126:34461 requested a full tablet report, sending...
I20260929 20:22:59.995352 22323 ts_manager.cc:194] Registered new tserver with Master: 0739d064d48a4eab8330d8df4a962f77 (127.17.53.67:33137)
I20260929 20:23:00.002648 17621 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 20:23:00.009858 22511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:00.007374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40498 (local address 127.17.53.126:34461)
0929 20:23:00.007541 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0929 20:23:00.007545 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:23:00.008104 (+ 559us) server_negotiation.cc:407] Connection header received
0929 20:23:00.008251 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:23:00.008254 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:23:00.008307 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:23:00.008399 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:23:00.009376 (+ 977us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:23:00.009380 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:23:00.009383 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:23:00.009396 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:23:00.009426 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:23:00.009437 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:23:00.009534 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0929 20:23:00.009573 (+ 39us) server_negotiation.cc:299] Negotiation successful
0929 20:23:00.009647 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 20:23:00.014151 22323 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:40498:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 20:23:00.021003 22323 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 20:23:00.055012 22782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:00.032891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.66:39471 (local address 127.0.0.1:41378)
0929 20:23:00.047563 (+ 14672us) negotiation.cc:107] Waiting for socket to connect
0929 20:23:00.047635 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0929 20:23:00.047831 (+ 196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:23:00.052655 (+ 4824us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:23:00.052663 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:23:00.052693 (+ 30us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:23:00.053603 (+ 910us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 20:23:00.053606 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:23:00.053639 (+ 33us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 20:23:00.053728 (+ 89us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:23:00.054167 (+ 439us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:23:00.054178 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:23:00.054185 (+ 7us) client_negotiation.cc:769] Sending connection context
0929 20:23:00.054340 (+ 155us) client_negotiation.cc:240] Negotiation successful
0929 20:23:00.054435 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14293,"thread_start_us":15084,"threads_started":1}
I20260929 20:23:00.055171 22785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:00.033475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.65:45127 (local address 127.0.0.1:33038)
0929 20:23:00.051420 (+ 17945us) negotiation.cc:107] Waiting for socket to connect
0929 20:23:00.051441 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 20:23:00.051559 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:23:00.053260 (+ 1701us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:23:00.053265 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:23:00.053296 (+ 31us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:23:00.053979 (+ 683us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 20:23:00.053982 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:23:00.053997 (+ 15us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 20:23:00.054036 (+ 39us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:23:00.054515 (+ 479us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:23:00.054520 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:23:00.054525 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 20:23:00.054680 (+ 155us) client_negotiation.cc:240] Negotiation successful
0929 20:23:00.054727 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17807,"thread_start_us":17655,"threads_started":1}
I20260929 20:23:00.055330 22787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:00.034016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33038 (local address 127.17.53.65:45127)
0929 20:23:00.050946 (+ 16930us) server_negotiation.cc:206] Beginning negotiation
0929 20:23:00.050957 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 20:23:00.051677 (+ 720us) server_negotiation.cc:407] Connection header received
0929 20:23:00.051750 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:23:00.051758 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:23:00.052041 (+ 283us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:23:00.052205 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:23:00.054217 (+ 2012us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:23:00.054225 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:23:00.054237 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:23:00.054266 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:23:00.054325 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:23:00.054346 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:23:00.054467 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0929 20:23:00.054704 (+ 237us) server_negotiation.cc:299] Negotiation successful
0929 20:23:00.054818 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16730,"thread_start_us":16607,"threads_started":1}
I20260929 20:23:00.055882 22784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:00.033175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41378 (local address 127.17.53.66:39471)
0929 20:23:00.052174 (+ 18999us) server_negotiation.cc:206] Beginning negotiation
0929 20:23:00.052182 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 20:23:00.052208 (+ 26us) server_negotiation.cc:407] Connection header received
0929 20:23:00.052274 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:23:00.052282 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:23:00.052451 (+ 169us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:23:00.052812 (+ 361us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:23:00.053904 (+ 1092us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:23:00.053916 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:23:00.053935 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:23:00.053961 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:23:00.054018 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:23:00.054037 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:23:00.055389 (+ 1352us) server_negotiation.cc:1035] Waiting for connection context
0929 20:23:00.055464 (+ 75us) server_negotiation.cc:299] Negotiation successful
0929 20:23:00.055585 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18832,"thread_start_us":20339,"threads_started":1}
I20260929 20:23:00.057217 22783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:00.032998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.67:33137 (local address 127.0.0.1:46538)
0929 20:23:00.050259 (+ 17261us) negotiation.cc:107] Waiting for socket to connect
0929 20:23:00.050283 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0929 20:23:00.050404 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:23:00.052588 (+ 2184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:23:00.052598 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:23:00.052688 (+ 90us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:23:00.054164 (+ 1476us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 20:23:00.054166 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:23:00.054181 (+ 15us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 20:23:00.054217 (+ 36us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:23:00.055894 (+ 1677us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:23:00.055899 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:23:00.055903 (+ 4us) client_negotiation.cc:769] Sending connection context
0929 20:23:00.056374 (+ 471us) client_negotiation.cc:240] Negotiation successful
0929 20:23:00.056426 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17109,"thread_start_us":16956,"threads_started":1}
I20260929 20:23:00.057379 22786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:00.033810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46538 (local address 127.17.53.67:33137)
0929 20:23:00.051958 (+ 18148us) server_negotiation.cc:206] Beginning negotiation
0929 20:23:00.051966 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 20:23:00.052001 (+ 35us) server_negotiation.cc:407] Connection header received
0929 20:23:00.052072 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:23:00.052079 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:23:00.052286 (+ 207us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:23:00.054009 (+ 1723us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:23:00.055710 (+ 1701us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:23:00.055716 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:23:00.055724 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:23:00.055736 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:23:00.055774 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:23:00.055787 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:23:00.055971 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0929 20:23:00.056915 (+ 944us) server_negotiation.cc:299] Negotiation successful
0929 20:23:00.057007 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17977,"thread_start_us":17769,"threads_started":1}
I20260929 20:23:00.059446 22444 tablet_service.cc:1467] Processing CreateTablet for tablet a2f890b5edc24e56bdb94b3d29d9ad34 (DEFAULT_TABLE table=test-table [id=3a332d02836b4ef4ab81c8f2b8925919]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:23:00.059441 22575 tablet_service.cc:1467] Processing CreateTablet for tablet a2f890b5edc24e56bdb94b3d29d9ad34 (DEFAULT_TABLE table=test-table [id=3a332d02836b4ef4ab81c8f2b8925919]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:23:00.060747 22444 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a2f890b5edc24e56bdb94b3d29d9ad34. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:23:00.060797 22575 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a2f890b5edc24e56bdb94b3d29d9ad34. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:23:00.060864 22705 tablet_service.cc:1467] Processing CreateTablet for tablet a2f890b5edc24e56bdb94b3d29d9ad34 (DEFAULT_TABLE table=test-table [id=3a332d02836b4ef4ab81c8f2b8925919]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:23:00.062124 22705 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a2f890b5edc24e56bdb94b3d29d9ad34. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:23:00.073336 22788 tablet_bootstrap.cc:492] T a2f890b5edc24e56bdb94b3d29d9ad34 P 2fff9d527a4e458999850bbb5c1f1412: Bootstrap starting.
I20260929 20:23:00.075554 22789 tablet_bootstrap.cc:492] T a2f890b5edc24e56bdb94b3d29d9ad34 P c6eda6e706d1440093d98981edeeee80: Bootstrap starting.
I20260929 20:23:00.076639 22788 tablet_bootstrap.cc:654] T a2f890b5edc24e56bdb94b3d29d9ad34 P 2fff9d527a4e458999850bbb5c1f1412: Neither blocks nor log segments found. Creating new log.
I20260929 20:23:00.077256 22790 tablet_bootstrap.cc:492] T a2f890b5edc24e56bdb94b3d29d9ad34 P 0739d064d48a4eab8330d8df4a962f77: Bootstrap starting.
I20260929 20:23:00.078011 22788 log.cc:824] T a2f890b5edc24e56bdb94b3d29d9ad34 P 2fff9d527a4e458999850bbb5c1f1412: Log is configured to *not* fsync() on all Append() calls
I20260929 20:23:00.079227 22789 tablet_bootstrap.cc:654] T a2f890b5edc24e56bdb94b3d29d9ad34 P c6eda6e706d1440093d98981edeeee80: Neither blocks nor log segments found. Creating new log.
I20260929 20:23:00.080456 22789 log.cc:824] T a2f890b5edc24e56bdb94b3d29d9ad34 P c6eda6e706d1440093d98981edeeee80: Log is configured to *not* fsync() on all Append() calls
I20260929 20:23:00.080549 22790 tablet_bootstrap.cc:654] T a2f890b5edc24e56bdb94b3d29d9ad34 P 0739d064d48a4eab8330d8df4a962f77: Neither blocks nor log segments found. Creating new log.
I20260929 20:23:00.081815 22790 log.cc:824] T a2f890b5edc24e56bdb94b3d29d9ad34 P 0739d064d48a4eab8330d8df4a962f77: Log is configured to *not* fsync() on all Append() calls
I20260929 20:23:00.082002 22788 tablet_bootstrap.cc:492] T a2f890b5edc24e56bdb94b3d29d9ad34 P 2fff9d527a4e458999850bbb5c1f1412: No bootstrap required, opened a new log
I20260929 20:23:00.082283 22788 ts_tablet_manager.cc:1397] T a2f890b5edc24e56bdb94b3d29d9ad34 P 2fff9d527a4e458999850bbb5c1f1412: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260929 20:23:00.083457 22789 tablet_bootstrap.cc:492] T a2f890b5edc24e56bdb94b3d29d9ad34 P c6eda6e706d1440093d98981edeeee80: No bootstrap required, opened a new log
I20260929 20:23:00.083720 22789 ts_tablet_manager.cc:1397] T a2f890b5edc24e56bdb94b3d29d9ad34 P c6eda6e706d1440093d98981edeeee80: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260929 20:23:00.084378 22790 tablet_bootstrap.cc:492] T a2f890b5edc24e56bdb94b3d29d9ad34 P 0739d064d48a4eab8330d8df4a962f77: No bootstrap required, opened a new log
I20260929 20:23:00.084630 22790 ts_tablet_manager.cc:1397] T a2f890b5edc24e56bdb94b3d29d9ad34 P 0739d064d48a4eab8330d8df4a962f77: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260929 20:23:00.091815 22790 raft_consensus.cc:348] T a2f890b5edc24e56bdb94b3d29d9ad34 P 0739d064d48a4eab8330d8df4a962f77 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6eda6e706d1440093d98981edeeee80" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 39471 } } peers { permanent_uuid: "0739d064d48a4eab8330d8df4a962f77" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 33137 } } peers { permanent_uuid: "2fff9d527a4e458999850bbb5c1f1412" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 45127 } }
I20260929 20:23:00.092226 22790 raft_consensus.cc:374] T a2f890b5edc24e56bdb94b3d29d9ad34 P 0739d064d48a4eab8330d8df4a962f77 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:23:00.092320 22790 raft_consensus.cc:729] T a2f890b5edc24e56bdb94b3d29d9ad34 P 0739d064d48a4eab8330d8df4a962f77 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0739d064d48a4eab8330d8df4a962f77, State: Initialized, Role: FOLLOWER
I20260929 20:23:00.092949 22790 consensus_queue.cc:260] T a2f890b5edc24e56bdb94b3d29d9ad34 P 0739d064d48a4eab8330d8df4a962f77 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6eda6e706d1440093d98981edeeee80" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 39471 } } peers { permanent_uuid: "0739d064d48a4eab8330d8df4a962f77" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 33137 } } peers { permanent_uuid: "2fff9d527a4e458999850bbb5c1f1412" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 45127 } }
I20260929 20:23:00.093523 22788 raft_consensus.cc:348] T a2f890b5edc24e56bdb94b3d29d9ad34 P 2fff9d527a4e458999850bbb5c1f1412 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6eda6e706d1440093d98981edeeee80" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 39471 } } peers { permanent_uuid: "0739d064d48a4eab8330d8df4a962f77" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 33137 } } peers { permanent_uuid: "2fff9d527a4e458999850bbb5c1f1412" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 45127 } }
I20260929 20:23:00.094247 22770 heartbeater.cc:503] Master 127.17.53.126:34461 was elected leader, sending a full tablet report...
I20260929 20:23:00.094142 22789 raft_consensus.cc:348] T a2f890b5edc24e56bdb94b3d29d9ad34 P c6eda6e706d1440093d98981edeeee80 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6eda6e706d1440093d98981edeeee80" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 39471 } } peers { permanent_uuid: "0739d064d48a4eab8330d8df4a962f77" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 33137 } } peers { permanent_uuid: "2fff9d527a4e458999850bbb5c1f1412" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 45127 } }
I20260929 20:23:00.093976 22788 raft_consensus.cc:374] T a2f890b5edc24e56bdb94b3d29d9ad34 P 2fff9d527a4e458999850bbb5c1f1412 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:23:00.094552 22789 raft_consensus.cc:374] T a2f890b5edc24e56bdb94b3d29d9ad34 P c6eda6e706d1440093d98981edeeee80 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:23:00.094722 22789 raft_consensus.cc:729] T a2f890b5edc24e56bdb94b3d29d9ad34 P c6eda6e706d1440093d98981edeeee80 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6eda6e706d1440093d98981edeeee80, State: Initialized, Role: FOLLOWER
I20260929 20:23:00.094903 22790 ts_tablet_manager.cc:1428] T a2f890b5edc24e56bdb94b3d29d9ad34 P 0739d064d48a4eab8330d8df4a962f77: Time spent starting tablet: real 0.010s user 0.005s sys 0.006s
I20260929 20:23:00.094904 22788 raft_consensus.cc:729] T a2f890b5edc24e56bdb94b3d29d9ad34 P 2fff9d527a4e458999850bbb5c1f1412 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2fff9d527a4e458999850bbb5c1f1412, State: Initialized, Role: FOLLOWER
I20260929 20:23:00.095413 22789 consensus_queue.cc:260] T a2f890b5edc24e56bdb94b3d29d9ad34 P c6eda6e706d1440093d98981edeeee80 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6eda6e706d1440093d98981edeeee80" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 39471 } } peers { permanent_uuid: "0739d064d48a4eab8330d8df4a962f77" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 33137 } } peers { permanent_uuid: "2fff9d527a4e458999850bbb5c1f1412" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 45127 } }
I20260929 20:23:00.095618 22788 consensus_queue.cc:260] T a2f890b5edc24e56bdb94b3d29d9ad34 P 2fff9d527a4e458999850bbb5c1f1412 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6eda6e706d1440093d98981edeeee80" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 39471 } } peers { permanent_uuid: "0739d064d48a4eab8330d8df4a962f77" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 33137 } } peers { permanent_uuid: "2fff9d527a4e458999850bbb5c1f1412" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 45127 } }
I20260929 20:23:00.096691 22640 heartbeater.cc:503] Master 127.17.53.126:34461 was elected leader, sending a full tablet report...
I20260929 20:23:00.097326 22789 ts_tablet_manager.cc:1428] T a2f890b5edc24e56bdb94b3d29d9ad34 P c6eda6e706d1440093d98981edeeee80: Time spent starting tablet: real 0.013s user 0.013s sys 0.000s
I20260929 20:23:00.098160 22509 heartbeater.cc:503] Master 127.17.53.126:34461 was elected leader, sending a full tablet report...
I20260929 20:23:00.098913 22788 ts_tablet_manager.cc:1428] T a2f890b5edc24e56bdb94b3d29d9ad34 P 2fff9d527a4e458999850bbb5c1f1412: Time spent starting tablet: real 0.016s user 0.011s sys 0.004s
W20260929 20:23:00.103611 22510 tablet.cc:2369] T a2f890b5edc24e56bdb94b3d29d9ad34 P 2fff9d527a4e458999850bbb5c1f1412: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 20:23:00.114462 22796 raft_consensus.cc:482] T a2f890b5edc24e56bdb94b3d29d9ad34 P 2fff9d527a4e458999850bbb5c1f1412 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 20:23:00.114996 22796 raft_consensus.cc:504] T a2f890b5edc24e56bdb94b3d29d9ad34 P 2fff9d527a4e458999850bbb5c1f1412 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6eda6e706d1440093d98981edeeee80" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 39471 } } peers { permanent_uuid: "0739d064d48a4eab8330d8df4a962f77" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 33137 } } peers { permanent_uuid: "2fff9d527a4e458999850bbb5c1f1412" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 45127 } }
I20260929 20:23:00.116878 22796 leader_election.cc:288] T a2f890b5edc24e56bdb94b3d29d9ad34 P 2fff9d527a4e458999850bbb5c1f1412 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c6eda6e706d1440093d98981edeeee80 (127.17.53.66:39471), 0739d064d48a4eab8330d8df4a962f77 (127.17.53.67:33137)
I20260929 20:23:00.120457 22784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:00.117269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.53.65:52753 (local address 127.17.53.66:39471)
0929 20:23:00.117523 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0929 20:23:00.117532 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 20:23:00.118396 (+ 864us) server_negotiation.cc:407] Connection header received
0929 20:23:00.118509 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:23:00.118518 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:23:00.118592 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:23:00.118701 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:23:00.119778 (+ 1077us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:23:00.119784 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:23:00.119789 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:23:00.119809 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:23:00.119844 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:23:00.119860 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:23:00.119975 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0929 20:23:00.120073 (+ 98us) server_negotiation.cc:299] Negotiation successful
0929 20:23:00.120118 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260929 20:23:00.120643 22798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:00.117212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.66:39471 (local address 127.17.53.65:52753)
0929 20:23:00.118279 (+ 1067us) negotiation.cc:107] Waiting for socket to connect
0929 20:23:00.118294 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 20:23:00.118393 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:23:00.118741 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:23:00.118746 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:23:00.118770 (+ 24us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:23:00.119597 (+ 827us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 20:23:00.119599 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:23:00.119612 (+ 13us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 20:23:00.119647 (+ 35us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:23:00.119960 (+ 313us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:23:00.119964 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:23:00.119969 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 20:23:00.120130 (+ 161us) client_negotiation.cc:240] Negotiation successful
0929 20:23:00.120186 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":927,"thread_start_us":732,"threads_started":1}
I20260929 20:23:00.120636 22786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:00.117411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.53.65:43613 (local address 127.17.53.67:33137)
0929 20:23:00.117617 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0929 20:23:00.117623 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:23:00.118581 (+ 958us) server_negotiation.cc:407] Connection header received
0929 20:23:00.118747 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:23:00.118752 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:23:00.118808 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:23:00.118939 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:23:00.119989 (+ 1050us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:23:00.119997 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:23:00.120004 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:23:00.120027 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:23:00.120069 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:23:00.120087 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:23:00.120202 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0929 20:23:00.120329 (+ 127us) server_negotiation.cc:299] Negotiation successful
0929 20:23:00.120388 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260929 20:23:00.121253 22797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:00.117370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.67:33137 (local address 127.17.53.65:43613)
0929 20:23:00.118417 (+ 1047us) negotiation.cc:107] Waiting for socket to connect
0929 20:23:00.118438 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 20:23:00.118600 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:23:00.118997 (+ 397us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:23:00.119001 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:23:00.119018 (+ 17us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:23:00.119778 (+ 760us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 20:23:00.119782 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:23:00.119797 (+ 15us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 20:23:00.119837 (+ 40us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:23:00.120187 (+ 350us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:23:00.120191 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:23:00.120196 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 20:23:00.120359 (+ 163us) client_negotiation.cc:240] Negotiation successful
0929 20:23:00.120404 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":906,"thread_start_us":473,"threads_started":1}
I20260929 20:23:00.121734 22595 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2f890b5edc24e56bdb94b3d29d9ad34" candidate_uuid: "2fff9d527a4e458999850bbb5c1f1412" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6eda6e706d1440093d98981edeeee80" is_pre_election: true
I20260929 20:23:00.122275 22595 raft_consensus.cc:2454] T a2f890b5edc24e56bdb94b3d29d9ad34 P c6eda6e706d1440093d98981edeeee80 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2fff9d527a4e458999850bbb5c1f1412 in term 0.
I20260929 20:23:00.122224 22725 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2f890b5edc24e56bdb94b3d29d9ad34" candidate_uuid: "2fff9d527a4e458999850bbb5c1f1412" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0739d064d48a4eab8330d8df4a962f77" is_pre_election: true
I20260929 20:23:00.122697 22725 raft_consensus.cc:2454] T a2f890b5edc24e56bdb94b3d29d9ad34 P 0739d064d48a4eab8330d8df4a962f77 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2fff9d527a4e458999850bbb5c1f1412 in term 0.
I20260929 20:23:00.122969 22399 leader_election.cc:302] T a2f890b5edc24e56bdb94b3d29d9ad34 P 2fff9d527a4e458999850bbb5c1f1412 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2fff9d527a4e458999850bbb5c1f1412, c6eda6e706d1440093d98981edeeee80; no voters:
I20260929 20:23:00.123448 22796 raft_consensus.cc:2781] T a2f890b5edc24e56bdb94b3d29d9ad34 P 2fff9d527a4e458999850bbb5c1f1412 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 20:23:00.123594 22796 raft_consensus.cc:482] T a2f890b5edc24e56bdb94b3d29d9ad34 P 2fff9d527a4e458999850bbb5c1f1412 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 20:23:00.123694 22796 raft_consensus.cc:3037] T a2f890b5edc24e56bdb94b3d29d9ad34 P 2fff9d527a4e458999850bbb5c1f1412 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:23:00.125792 22796 raft_consensus.cc:504] T a2f890b5edc24e56bdb94b3d29d9ad34 P 2fff9d527a4e458999850bbb5c1f1412 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6eda6e706d1440093d98981edeeee80" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 39471 } } peers { permanent_uuid: "0739d064d48a4eab8330d8df4a962f77" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 33137 } } peers { permanent_uuid: "2fff9d527a4e458999850bbb5c1f1412" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 45127 } }
I20260929 20:23:00.126415 22796 leader_election.cc:288] T a2f890b5edc24e56bdb94b3d29d9ad34 P 2fff9d527a4e458999850bbb5c1f1412 [CANDIDATE]: Term 1 election: Requested vote from peers c6eda6e706d1440093d98981edeeee80 (127.17.53.66:39471), 0739d064d48a4eab8330d8df4a962f77 (127.17.53.67:33137)
I20260929 20:23:00.126782 22595 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2f890b5edc24e56bdb94b3d29d9ad34" candidate_uuid: "2fff9d527a4e458999850bbb5c1f1412" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6eda6e706d1440093d98981edeeee80"
I20260929 20:23:00.126996 22725 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2f890b5edc24e56bdb94b3d29d9ad34" candidate_uuid: "2fff9d527a4e458999850bbb5c1f1412" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0739d064d48a4eab8330d8df4a962f77"
I20260929 20:23:00.127034 22595 raft_consensus.cc:3037] T a2f890b5edc24e56bdb94b3d29d9ad34 P c6eda6e706d1440093d98981edeeee80 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:23:00.127235 22725 raft_consensus.cc:3037] T a2f890b5edc24e56bdb94b3d29d9ad34 P 0739d064d48a4eab8330d8df4a962f77 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:23:00.129796 22595 raft_consensus.cc:2454] T a2f890b5edc24e56bdb94b3d29d9ad34 P c6eda6e706d1440093d98981edeeee80 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2fff9d527a4e458999850bbb5c1f1412 in term 1.
I20260929 20:23:00.130043 22725 raft_consensus.cc:2454] T a2f890b5edc24e56bdb94b3d29d9ad34 P 0739d064d48a4eab8330d8df4a962f77 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2fff9d527a4e458999850bbb5c1f1412 in term 1.
I20260929 20:23:00.130226 22399 leader_election.cc:302] T a2f890b5edc24e56bdb94b3d29d9ad34 P 2fff9d527a4e458999850bbb5c1f1412 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2fff9d527a4e458999850bbb5c1f1412, c6eda6e706d1440093d98981edeeee80; no voters:
I20260929 20:23:00.130546 22796 raft_consensus.cc:2781] T a2f890b5edc24e56bdb94b3d29d9ad34 P 2fff9d527a4e458999850bbb5c1f1412 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:23:00.131043 22796 raft_consensus.cc:686] T a2f890b5edc24e56bdb94b3d29d9ad34 P 2fff9d527a4e458999850bbb5c1f1412 [term 1 LEADER]: Becoming Leader. State: Replica: 2fff9d527a4e458999850bbb5c1f1412, State: Running, Role: LEADER
I20260929 20:23:00.131651 22796 consensus_queue.cc:237] T a2f890b5edc24e56bdb94b3d29d9ad34 P 2fff9d527a4e458999850bbb5c1f1412 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6eda6e706d1440093d98981edeeee80" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 39471 } } peers { permanent_uuid: "0739d064d48a4eab8330d8df4a962f77" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 33137 } } peers { permanent_uuid: "2fff9d527a4e458999850bbb5c1f1412" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 45127 } }
I20260929 20:23:00.136547 22321 catalog_manager.cc:5495] T a2f890b5edc24e56bdb94b3d29d9ad34 P 2fff9d527a4e458999850bbb5c1f1412 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2fff9d527a4e458999850bbb5c1f1412 (127.17.53.65). New cstate: current_term: 1 leader_uuid: "2fff9d527a4e458999850bbb5c1f1412" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6eda6e706d1440093d98981edeeee80" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 39471 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0739d064d48a4eab8330d8df4a962f77" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 33137 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2fff9d527a4e458999850bbb5c1f1412" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 45127 } health_report { overall_health: HEALTHY } } }
I20260929 20:23:00.157444 22787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:00.155439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33054 (local address 127.17.53.65:45127)
0929 20:23:00.155648 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0929 20:23:00.155654 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:23:00.155675 (+ 21us) server_negotiation.cc:407] Connection header received
0929 20:23:00.155878 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:23:00.155882 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:23:00.155961 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:23:00.156043 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:23:00.156837 (+ 794us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:23:00.156841 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:23:00.156844 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:23:00.156859 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:23:00.156887 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:23:00.156899 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:23:00.156988 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0929 20:23:00.157070 (+ 82us) server_negotiation.cc:299] Negotiation successful
0929 20:23:00.157131 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260929 20:23:00.165871 22595 raft_consensus.cc:1261] T a2f890b5edc24e56bdb94b3d29d9ad34 P c6eda6e706d1440093d98981edeeee80 [term 1 FOLLOWER]: Refusing update from remote peer 2fff9d527a4e458999850bbb5c1f1412: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:23:00.165871 22725 raft_consensus.cc:1261] T a2f890b5edc24e56bdb94b3d29d9ad34 P 0739d064d48a4eab8330d8df4a962f77 [term 1 FOLLOWER]: Refusing update from remote peer 2fff9d527a4e458999850bbb5c1f1412: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:23:00.166891 22799 consensus_queue.cc:1035] T a2f890b5edc24e56bdb94b3d29d9ad34 P 2fff9d527a4e458999850bbb5c1f1412 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0739d064d48a4eab8330d8df4a962f77" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 33137 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:23:00.167254 22796 consensus_queue.cc:1035] T a2f890b5edc24e56bdb94b3d29d9ad34 P 2fff9d527a4e458999850bbb5c1f1412 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6eda6e706d1440093d98981edeeee80" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 39471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:23:00.176606 22802 mvcc.cc:205] Tried to move back new op lower bound from 7334762005150068736 to 7334762005023281152. Current Snapshot: MvccSnapshot[applied={T|T < 7334762005150068736}]
I20260929 20:23:00.178341 22803 mvcc.cc:205] Tried to move back new op lower bound from 7334762005150068736 to 7334762005023281152. Current Snapshot: MvccSnapshot[applied={T|T < 7334762005150068736}]
I20260929 20:23:00.183730 22804 mvcc.cc:205] Tried to move back new op lower bound from 7334762005150068736 to 7334762005023281152. Current Snapshot: MvccSnapshot[applied={T|T < 7334762005150068736}]
I20260929 20:23:00.194377 22321 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:40498:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 20:23:00.194757 22321 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:40498:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 20:23:00.198361 22321 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7a41f7c7baf64f50a72a85c232bc81c0: Sending DeleteTablet for 3 replicas of tablet a2f890b5edc24e56bdb94b3d29d9ad34
I20260929 20:23:00.199575 22705 tablet_service.cc:1514] Processing DeleteTablet for tablet a2f890b5edc24e56bdb94b3d29d9ad34 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:23:00 UTC) from {username='slave'} at 127.0.0.1:46538
I20260929 20:23:00.199645 22575 tablet_service.cc:1514] Processing DeleteTablet for tablet a2f890b5edc24e56bdb94b3d29d9ad34 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:23:00 UTC) from {username='slave'} at 127.0.0.1:41378
I20260929 20:23:00.199707 22444 tablet_service.cc:1514] Processing DeleteTablet for tablet a2f890b5edc24e56bdb94b3d29d9ad34 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:23:00 UTC) from {username='slave'} at 127.0.0.1:33038
I20260929 20:23:00.202010 17621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu pid 22382
I20260929 20:23:00.210282 22813 tablet_replica.cc:330] T a2f890b5edc24e56bdb94b3d29d9ad34 P c6eda6e706d1440093d98981edeeee80: stopping tablet replica
I20260929 20:23:00.211741 22813 raft_consensus.cc:2229] T a2f890b5edc24e56bdb94b3d29d9ad34 P c6eda6e706d1440093d98981edeeee80 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 20:23:00.212491 22813 raft_consensus.cc:2258] T a2f890b5edc24e56bdb94b3d29d9ad34 P c6eda6e706d1440093d98981edeeee80 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:23:00.213749 22814 tablet_replica.cc:330] T a2f890b5edc24e56bdb94b3d29d9ad34 P 2fff9d527a4e458999850bbb5c1f1412: stopping tablet replica
I20260929 20:23:00.214170 22814 raft_consensus.cc:2229] T a2f890b5edc24e56bdb94b3d29d9ad34 P 2fff9d527a4e458999850bbb5c1f1412 [term 1 LEADER]: Raft consensus shutting down.
I20260929 20:23:00.214926 22814 raft_consensus.cc:2258] T a2f890b5edc24e56bdb94b3d29d9ad34 P 2fff9d527a4e458999850bbb5c1f1412 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:23:00.217453 22812 tablet_replica.cc:330] T a2f890b5edc24e56bdb94b3d29d9ad34 P 0739d064d48a4eab8330d8df4a962f77: stopping tablet replica
I20260929 20:23:00.217612 22813 ts_tablet_manager.cc:1905] T a2f890b5edc24e56bdb94b3d29d9ad34 P c6eda6e706d1440093d98981edeeee80: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:23:00.217947 22812 raft_consensus.cc:2229] T a2f890b5edc24e56bdb94b3d29d9ad34 P 0739d064d48a4eab8330d8df4a962f77 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 20:23:00.218515 22812 raft_consensus.cc:2258] T a2f890b5edc24e56bdb94b3d29d9ad34 P 0739d064d48a4eab8330d8df4a962f77 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:23:00.219365 22787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:00.203682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33058 (local address 127.17.53.65:45127)
0929 20:23:00.203910 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0929 20:23:00.203919 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 20:23:00.216618 (+ 12699us) server_negotiation.cc:407] Connection header received
0929 20:23:00.216838 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:23:00.216844 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:23:00.216905 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:23:00.217019 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:23:00.218431 (+ 1412us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:23:00.218437 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:23:00.218446 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:23:00.218468 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:23:00.218513 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:23:00.218531 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:23:00.218634 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0929 20:23:00.219023 (+ 389us) server_negotiation.cc:299] Negotiation successful
0929 20:23:00.219086 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260929 20:23:00.220451 22504 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:23:00.220911 22812 ts_tablet_manager.cc:1905] T a2f890b5edc24e56bdb94b3d29d9ad34 P 0739d064d48a4eab8330d8df4a962f77: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:23:00.226127 22813 ts_tablet_manager.cc:1918] T a2f890b5edc24e56bdb94b3d29d9ad34 P c6eda6e706d1440093d98981edeeee80: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 20:23:00.226326 22813 log.cc:1196] T a2f890b5edc24e56bdb94b3d29d9ad34 P c6eda6e706d1440093d98981edeeee80: Deleting WAL directory at /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790713339161492-17621-0/minicluster-data/ts-1/wal/wals/a2f890b5edc24e56bdb94b3d29d9ad34
I20260929 20:23:00.226764 22813 ts_tablet_manager.cc:1939] T a2f890b5edc24e56bdb94b3d29d9ad34 P c6eda6e706d1440093d98981edeeee80: Deleting consensus metadata
I20260929 20:23:00.228363 22309 catalog_manager.cc:4841] TS c6eda6e706d1440093d98981edeeee80 (127.17.53.66:39471): tablet a2f890b5edc24e56bdb94b3d29d9ad34 (table test-table [id=3a332d02836b4ef4ab81c8f2b8925919]) successfully deleted
I20260929 20:23:00.228580 22812 ts_tablet_manager.cc:1918] T a2f890b5edc24e56bdb94b3d29d9ad34 P 0739d064d48a4eab8330d8df4a962f77: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 20:23:00.228734 22812 log.cc:1196] T a2f890b5edc24e56bdb94b3d29d9ad34 P 0739d064d48a4eab8330d8df4a962f77: Deleting WAL directory at /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790713339161492-17621-0/minicluster-data/ts-2/wal/wals/a2f890b5edc24e56bdb94b3d29d9ad34
I20260929 20:23:00.229261 22812 ts_tablet_manager.cc:1939] T a2f890b5edc24e56bdb94b3d29d9ad34 P 0739d064d48a4eab8330d8df4a962f77: Deleting consensus metadata
I20260929 20:23:00.231006 22307 catalog_manager.cc:4841] TS 0739d064d48a4eab8330d8df4a962f77 (127.17.53.67:33137): tablet a2f890b5edc24e56bdb94b3d29d9ad34 (table test-table [id=3a332d02836b4ef4ab81c8f2b8925919]) successfully deleted
I20260929 20:23:00.316622 17621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu with pid 22382
W20260929 20:23:00.334527 22308 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.17.53.65:45127 (error 108)
I20260929 20:23:00.335330 17621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu pid 22513
I20260929 20:23:00.336511 22783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:00.335987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.65:45127 (local address 127.0.0.1:36550)
0929 20:23:00.336192 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0929 20:23:00.336328 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.53.65:45127: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260929 20:23:00.336735 22308 catalog_manager.cc:4584] TS 2fff9d527a4e458999850bbb5c1f1412 (127.17.53.65:45127): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a2f890b5edc24e56bdb94b3d29d9ad34: Network error: Client connection negotiation failed: client connection to 127.17.53.65:45127: connect: Connection refused (error 111)
I20260929 20:23:00.338239 22784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:00.336163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33180 (local address 127.17.53.66:39471)
0929 20:23:00.336339 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0929 20:23:00.336343 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:23:00.336406 (+ 63us) server_negotiation.cc:407] Connection header received
0929 20:23:00.336635 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:23:00.336640 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:23:00.336742 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:23:00.336834 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:23:00.337753 (+ 919us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:23:00.337757 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:23:00.337760 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:23:00.337773 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:23:00.337799 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:23:00.337810 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:23:00.337904 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0929 20:23:00.337994 (+ 90us) server_negotiation.cc:299] Negotiation successful
0929 20:23:00.338037 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260929 20:23:00.339021 22635 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:23:00.390489 22783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:00.389975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.65:45127 (local address 127.0.0.1:36558)
0929 20:23:00.390235 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0929 20:23:00.390336 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.53.65:45127: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126}
I20260929 20:23:00.391682 22783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:00.391305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.65:45127 (local address 127.0.0.1:36570)
0929 20:23:00.391493 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0929 20:23:00.391568 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.53.65:45127: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260929 20:23:00.426470 17621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu with pid 22513
I20260929 20:23:00.443578 17621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu pid 22643
I20260929 20:23:00.447054 22786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:00.444320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56408 (local address 127.17.53.67:33137)
0929 20:23:00.444565 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0929 20:23:00.444572 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:23:00.444710 (+ 138us) server_negotiation.cc:407] Connection header received
0929 20:23:00.444970 (+ 260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:23:00.444975 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:23:00.445079 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:23:00.445207 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:23:00.446292 (+ 1085us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:23:00.446298 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:23:00.446309 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:23:00.446326 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:23:00.446361 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:23:00.446373 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:23:00.446519 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0929 20:23:00.446620 (+ 101us) server_negotiation.cc:299] Negotiation successful
0929 20:23:00.446686 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260929 20:23:00.447664 22765 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:23:00.452396 22783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:00.451941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.65:45127 (local address 127.0.0.1:36574)
0929 20:23:00.452121 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0929 20:23:00.452245 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.53.65:45127: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260929 20:23:00.453588 22783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:00.453143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.65:45127 (local address 127.0.0.1:36578)
0929 20:23:00.453317 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0929 20:23:00.453413 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.53.65:45127: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260929 20:23:00.534154 22783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:00.533652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.65:45127 (local address 127.0.0.1:36594)
0929 20:23:00.533885 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0929 20:23:00.533992 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.53.65:45127: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
I20260929 20:23:00.535490 22783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:00.534903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.65:45127 (local address 127.0.0.1:36606)
0929 20:23:00.535194 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0929 20:23:00.535342 (+ 148us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.53.65:45127: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116}
I20260929 20:23:00.539907 17621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu with pid 22643
I20260929 20:23:00.556993 17621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu pid 22292
I20260929 20:23:00.558223 22353 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:23:00.663121 17621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu with pid 22292
2026-09-29T20:23:00Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (2591 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260929 20:23:00.704505 17621 mini_oidc.cc:152] Starting JWKS server
I20260929 20:23:00.707075 17621 webserver.cc:460] Webserver started at https://127.0.0.1:42429/ using document root <none> and password file <none>
I20260929 20:23:00.707381 17621 mini_oidc.cc:189] Starting OIDC Discovery server
I20260929 20:23:00.707958 17621 webserver.cc:460] Webserver started at http://127.0.0.1:35675/ using document root <none> and password file <none>
2026-09-29T20:23:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T20:23:00Z Disabled control of system clock
I20260929 20:23:00.734378 17621 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
/tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790713339161492-17621-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790713339161492-17621-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790713339161492-17621-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790713339161492-17621-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.53.126:41171
--webserver_interface=127.17.53.126
--webserver_port=0
--builtin_ntp_servers=127.17.53.84:45107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.53.126:41171
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42429/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790713339161492-17621-0/testchainca.cert with env {}
W20260929 20:23:01.069301 22826 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:23:01.069581 22826 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:23:01.069669 22826 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:23:01.077258 22826 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:23:01.077371 22826 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:23:01.077410 22826 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:23:01.077471 22826 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:23:01.077513 22826 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260929 20:23:01.077575 22826 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260929 20:23:01.077615 22826 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:42429/jwks.json
I20260929 20:23:01.087512 22826 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.53.84:45107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790713339161492-17621-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790713339161492-17621-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.53.126:41171
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.53.126:41171
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42429/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790713339161492-17621-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.53.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790713339161492-17621-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790713339161492-17621-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:23:01.088711 22826 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:23:01.089929 22826 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:23:01.103454 22831 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:23:01.103665 22832 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:23:01.104053 22835 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:23:01.105116 22826 server_base.cc:1034] running on GCE node
I20260929 20:23:01.105901 22826 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:23:01.107522 22826 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:23:01.108728 22826 hybrid_clock.cc:648] HybridClock initialized: now 1790713381108668 us; error 64 us; skew 500 ppm
I20260929 20:23:01.109131 22826 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 20:23:01.112353 22826 webserver.cc:460] Webserver started at http://127.17.53.126:37989/ using document root <none> and password file <none>
I20260929 20:23:01.113397 22826 fs_manager.cc:359] Metadata directory not provided
I20260929 20:23:01.113552 22826 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:23:01.113890 22826 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:23:01.117889 22826 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790713339161492-17621-0/minicluster-data/master-0/data/instance:
uuid: "0d53ab1f536645378296399aefeedb4c"
format_stamp: "Formatted at 2026-09-29 20:23:01 on dist-test-slave-zbnc"
server_key: "842ee8e8c55593b57de52e6ccafc7783"
server_key_iv: "4f068796ecc44307e2295a18a3f5a1e0"
server_key_version: "encryptionkey@0"
I20260929 20:23:01.119164 22826 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790713339161492-17621-0/minicluster-data/master-0/wal/instance:
uuid: "0d53ab1f536645378296399aefeedb4c"
format_stamp: "Formatted at 2026-09-29 20:23:01 on dist-test-slave-zbnc"
server_key: "842ee8e8c55593b57de52e6ccafc7783"
server_key_iv: "4f068796ecc44307e2295a18a3f5a1e0"
server_key_version: "encryptionkey@0"
I20260929 20:23:01.127164 22826 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.003s sys 0.005s
I20260929 20:23:01.132182 22841 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:23:01.134390 22826 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260929 20:23:01.134613 22826 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790713339161492-17621-0/minicluster-data/master-0/data,/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790713339161492-17621-0/minicluster-data/master-0/wal
uuid: "0d53ab1f536645378296399aefeedb4c"
format_stamp: "Formatted at 2026-09-29 20:23:01 on dist-test-slave-zbnc"
server_key: "842ee8e8c55593b57de52e6ccafc7783"
server_key_iv: "4f068796ecc44307e2295a18a3f5a1e0"
server_key_version: "encryptionkey@0"
I20260929 20:23:01.134939 22826 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790713339161492-17621-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790713339161492-17621-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790713339161492-17621-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:23:01.170288 22826 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:23:01.171303 22826 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:23:01.171640 22826 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:23:01.191193 22826 rpc_server.cc:307] RPC server started. Bound to: 127.17.53.126:41171
I20260929 20:23:01.191215 22892 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.53.126:41171 every 8 connection(s)
I20260929 20:23:01.192694 22826 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790713339161492-17621-0/minicluster-data/master-0/data/info.pb
I20260929 20:23:01.193336 17621 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu as pid 22826
I20260929 20:23:01.193555 17621 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790713339161492-17621-0/minicluster-data/master-0/wal/instance
I20260929 20:23:01.194386 17621 external_mini_cluster.cc:1428] Setting key ae04c2c2ef7fb99f57cf0446e0d65da9
I20260929 20:23:01.197510 22893 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:23:01.208806 22896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:01.196470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46722 (local address 127.17.53.126:41171)
0929 20:23:01.197488 (+ 1018us) server_negotiation.cc:206] Beginning negotiation
0929 20:23:01.197504 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 20:23:01.197548 (+ 44us) server_negotiation.cc:407] Connection header received
0929 20:23:01.199350 (+ 1802us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:23:01.199371 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:23:01.199857 (+ 486us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:23:01.200216 (+ 359us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:23:01.200893 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:01.202236 (+ 1343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:01.204242 (+ 2006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:01.204898 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:01.205028 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:01.206348 (+ 1320us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:23:01.206355 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:23:01.206369 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:23:01.207023 (+ 654us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:23:01.207101 (+ 78us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:23:01.207121 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:23:01.207221 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0929 20:23:01.207453 (+ 232us) server_negotiation.cc:299] Negotiation successful
0929 20:23:01.207830 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":585,"thread_start_us":381,"threads_started":1}
I20260929 20:23:01.210402 22893 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d53ab1f536645378296399aefeedb4c: Bootstrap starting.
I20260929 20:23:01.212842 17621 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 20:23:01.213828 22893 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0d53ab1f536645378296399aefeedb4c: Neither blocks nor log segments found. Creating new log.
I20260929 20:23:01.214790 22893 log.cc:824] T 00000000000000000000000000000000 P 0d53ab1f536645378296399aefeedb4c: Log is configured to *not* fsync() on all Append() calls
I20260929 20:23:01.217661 22893 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d53ab1f536645378296399aefeedb4c: No bootstrap required, opened a new log
I20260929 20:23:01.225096 22893 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0d53ab1f536645378296399aefeedb4c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d53ab1f536645378296399aefeedb4c" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 41171 } }
I20260929 20:23:01.225508 22893 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0d53ab1f536645378296399aefeedb4c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:23:01.225615 22893 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0d53ab1f536645378296399aefeedb4c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d53ab1f536645378296399aefeedb4c, State: Initialized, Role: FOLLOWER
I20260929 20:23:01.226177 22893 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0d53ab1f536645378296399aefeedb4c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d53ab1f536645378296399aefeedb4c" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 41171 } }
I20260929 20:23:01.226428 22893 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0d53ab1f536645378296399aefeedb4c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 20:23:01.226569 22893 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0d53ab1f536645378296399aefeedb4c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 20:23:01.226742 22893 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0d53ab1f536645378296399aefeedb4c [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:23:01.229529 22893 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0d53ab1f536645378296399aefeedb4c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d53ab1f536645378296399aefeedb4c" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 41171 } }
I20260929 20:23:01.230026 22893 leader_election.cc:302] T 00000000000000000000000000000000 P 0d53ab1f536645378296399aefeedb4c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0d53ab1f536645378296399aefeedb4c; no voters:
I20260929 20:23:01.230598 22893 leader_election.cc:288] T 00000000000000000000000000000000 P 0d53ab1f536645378296399aefeedb4c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 20:23:01.230741 22898 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0d53ab1f536645378296399aefeedb4c [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:23:01.231523 22898 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0d53ab1f536645378296399aefeedb4c [term 1 LEADER]: Becoming Leader. State: Replica: 0d53ab1f536645378296399aefeedb4c, State: Running, Role: LEADER
I20260929 20:23:01.232023 22898 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0d53ab1f536645378296399aefeedb4c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d53ab1f536645378296399aefeedb4c" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 41171 } }
I20260929 20:23:01.232676 22893 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0d53ab1f536645378296399aefeedb4c [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:23:01.235000 22900 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d53ab1f536645378296399aefeedb4c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0d53ab1f536645378296399aefeedb4c. Latest consensus state: current_term: 1 leader_uuid: "0d53ab1f536645378296399aefeedb4c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d53ab1f536645378296399aefeedb4c" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 41171 } } }
I20260929 20:23:01.235430 22900 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d53ab1f536645378296399aefeedb4c [sys.catalog]: This master's current role is: LEADER
I20260929 20:23:01.235081 22899 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d53ab1f536645378296399aefeedb4c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0d53ab1f536645378296399aefeedb4c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d53ab1f536645378296399aefeedb4c" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 41171 } } }
I20260929 20:23:01.236083 22899 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d53ab1f536645378296399aefeedb4c [sys.catalog]: This master's current role is: LEADER
I20260929 20:23:01.238642 22907 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:23:01.243285 22907 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:23:01.251653 22907 catalog_manager.cc:1347] Generated new cluster ID: cf9eb0f6a5024f98bff094a67fc689a9
I20260929 20:23:01.251788 22907 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:23:01.286469 22907 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 20:23:01.287844 22907 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:23:01.301916 22907 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0d53ab1f536645378296399aefeedb4c: Generated new TSK 0
I20260929 20:23:01.302829 22907 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260929 20:23:01.339296 22923 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260929 20:23:01.339498 22923 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260929 20:23:01.340622 22896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:01.318751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46734 (local address 127.17.53.126:41171)
0929 20:23:01.318958 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0929 20:23:01.318965 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:23:01.319409 (+ 444us) server_negotiation.cc:407] Connection header received
0929 20:23:01.319703 (+ 294us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:23:01.319710 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:23:01.319804 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:23:01.319964 (+ 160us) server_negotiation.cc:226] Negotiated authn=JWT
0929 20:23:01.320474 (+ 510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:01.321642 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:01.323063 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:01.323772 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:01.323916 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:01.324421 (+ 505us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0929 20:23:01.339912 (+ 15491us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42429/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0929 20:23:01.340303 (+ 391us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:46734: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42429/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":69}
W20260929 20:23:01.340742 22896 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:46734: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42429/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260929 20:23:01.342276 22922 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:23:01.318614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:41171 (local address 127.0.0.1:46734)
0929 20:23:01.319303 (+ 689us) negotiation.cc:107] Waiting for socket to connect
0929 20:23:01.319320 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 20:23:01.319475 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:23:01.319990 (+ 515us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:23:01.319994 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:23:01.320004 (+ 10us) client_negotiation.cc:189] Negotiated authn=JWT
0929 20:23:01.320329 (+ 325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:23:01.320336 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:23:01.321776 (+ 1440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:23:01.321780 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:23:01.322923 (+ 1143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:23:01.322939 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:23:01.323934 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:23:01.323938 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:23:01.324059 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:01.324120 (+ 61us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0929 20:23:01.340271 (+ 16151us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42429/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0929 20:23:01.340596 (+ 325us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.17.53.126:41171: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42429/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":509,"thread_start_us":364,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.17.53.126:41171: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42429/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260929 20:23:01.344447 17621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu pid 22826
I20260929 20:23:01.345588 22888 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:23:01.440330 17621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu with pid 22826
2026-09-29T20:23:01Z chronyd exiting
I20260929 20:23:01.490434 17621 test_util.cc:175] -----------------------------------------------
I20260929 20:23:01.490569 17621 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790713339161492-17621-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (790 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260929 20:23:01.494143 17621 mini_oidc.cc:152] Starting JWKS server
I20260929 20:23:01.495996 17621 webserver.cc:460] Webserver started at https://127.0.0.1:46427/ using document root <none> and password file <none>
I20260929 20:23:01.496173 17621 mini_oidc.cc:189] Starting OIDC Discovery server
I20260929 20:23:01.496779 17621 webserver.cc:460] Webserver started at http://127.0.0.1:35065/ using document root <none> and password file <none>
2026-09-29T20:23:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T20:23:01Z Disabled control of system clock
I20260929 20:23:01.522533 17621 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
/tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790713339161492-17621-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790713339161492-17621-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790713339161492-17621-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790713339161492-17621-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.53.126:41093
--webserver_interface=127.17.53.126
--webserver_port=0
--builtin_ntp_servers=127.17.53.84:36817
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.53.126:41093
--enable_jwt_token_auth=true
--jwks_url=https://localhost:46427/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790713339161492-17621-0/testchainca.cert with env {}
W20260929 20:23:01.845762 22930 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:23:01.846119 22930 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:23:01.846297 22930 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:23:01.853674 22930 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:23:01.853827 22930 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:23:01.853917 22930 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:23:01.854028 22930 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:23:01.854112 22930 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260929 20:23:01.854202 22930 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260929 20:23:01.854281 22930 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:46427/jwks.json
I20260929 20:23:01.864267 22930 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.53.84:36817
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790713339161492-17621-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790713339161492-17621-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.53.126:41093
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.53.126:41093
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:46427/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790713339161492-17621-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.53.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790713339161492-17621-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790713339161492-17621-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:23:01.865605 22930 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:23:01.866935 22930 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:23:01.879142 22935 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:23:01.879212 22936 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:23:01.879287 22938 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:23:01.879927 22930 server_base.cc:1034] running on GCE node
I20260929 20:23:01.880869 22930 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:23:01.882355 22930 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:23:01.883590 22930 hybrid_clock.cc:648] HybridClock initialized: now 1790713381883547 us; error 45 us; skew 500 ppm
I20260929 20:23:01.883948 22930 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 20:23:01.886302 22930 webserver.cc:460] Webserver started at http://127.17.53.126:39127/ using document root <none> and password file <none>
I20260929 20:23:01.887063 22930 fs_manager.cc:359] Metadata directory not provided
I20260929 20:23:01.887207 22930 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:23:01.887529 22930 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:23:01.890414 22930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790713339161492-17621-0/minicluster-data/master-0/data/instance:
uuid: "1aaadc9b2b6b4653a121856fe4574880"
format_stamp: "Formatted at 2026-09-29 20:23:01 on dist-test-slave-zbnc"
server_key: "3b59779ae5cb8a0768816c01dd22d226"
server_key_iv: "68c7a71aed50b100db5fe1c5d1d8f2b6"
server_key_version: "encryptionkey@0"
I20260929 20:23:01.891287 22930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790713339161492-17621-0/minicluster-data/master-0/wal/instance:
uuid: "1aaadc9b2b6b4653a121856fe4574880"
format_stamp: "Formatted at 2026-09-29 20:23:01 on dist-test-slave-zbnc"
server_key: "3b59779ae5cb8a0768816c01dd22d226"
server_key_iv: "68c7a71aed50b100db5fe1c5d1d8f2b6"
server_key_version: "encryptionkey@0"
I20260929 20:23:01.896144 22930 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 20:23:01.899482 22944 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:23:01.900924 22930 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 20:23:01.901118 22930 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790713339161492-17621-0/minicluster-data/master-0/data,/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790713339161492-17621-0/minicluster-data/master-0/wal
uuid: "1aaadc9b2b6b4653a121856fe4574880"
format_stamp: "Formatted at 2026-09-29 20:23:01 on dist-test-slave-zbnc"
server_key: "3b59779ae5cb8a0768816c01dd22d226"
server_key_iv: "68c7a71aed50b100db5fe1c5d1d8f2b6"
server_key_version: "encryptionkey@0"
I20260929 20:23:01.901346 22930 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790713339161492-17621-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790713339161492-17621-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790713339161492-17621-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:23:01.915436 22930 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:23:01.916278 22930 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:23:01.916591 22930 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:23:01.935336 22930 rpc_server.cc:307] RPC server started. Bound to: 127.17.53.126:41093
I20260929 20:23:01.935343 22995 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.53.126:41093 every 8 connection(s)
I20260929 20:23:01.936738 22930 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790713339161492-17621-0/minicluster-data/master-0/data/info.pb
I20260929 20:23:01.940572 22996 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:23:01.940861 17621 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu as pid 22930
I20260929 20:23:01.941053 17621 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790713339161492-17621-0/minicluster-data/master-0/wal/instance
I20260929 20:23:01.941695 17621 external_mini_cluster.cc:1428] Setting key 11735db0cfe1a02d42ab462bf708f80c
I20260929 20:23:01.952644 22996 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1aaadc9b2b6b4653a121856fe4574880: Bootstrap starting.
I20260929 20:23:01.955039 22999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:01.943631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57880 (local address 127.17.53.126:41093)
0929 20:23:01.944617 (+ 986us) server_negotiation.cc:206] Beginning negotiation
0929 20:23:01.944635 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0929 20:23:01.944678 (+ 43us) server_negotiation.cc:407] Connection header received
0929 20:23:01.946493 (+ 1815us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:23:01.946514 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:23:01.947064 (+ 550us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:23:01.947449 (+ 385us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:23:01.947998 (+ 549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:01.949221 (+ 1223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:01.950545 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:01.951267 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:01.951455 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:01.952646 (+ 1191us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:23:01.952654 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:23:01.952666 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:23:01.953317 (+ 651us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:23:01.953395 (+ 78us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:23:01.953420 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:23:01.953543 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0929 20:23:01.953778 (+ 235us) server_negotiation.cc:299] Negotiation successful
0929 20:23:01.954067 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":560,"thread_start_us":348,"threads_started":1}
I20260929 20:23:01.956009 22996 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1aaadc9b2b6b4653a121856fe4574880: Neither blocks nor log segments found. Creating new log.
I20260929 20:23:01.957031 22996 log.cc:824] T 00000000000000000000000000000000 P 1aaadc9b2b6b4653a121856fe4574880: Log is configured to *not* fsync() on all Append() calls
I20260929 20:23:01.958743 17621 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 20:23:01.960143 22996 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1aaadc9b2b6b4653a121856fe4574880: No bootstrap required, opened a new log
I20260929 20:23:01.968237 22996 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1aaadc9b2b6b4653a121856fe4574880 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1aaadc9b2b6b4653a121856fe4574880" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 41093 } }
I20260929 20:23:01.968554 22996 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1aaadc9b2b6b4653a121856fe4574880 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:23:01.968632 22996 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1aaadc9b2b6b4653a121856fe4574880 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1aaadc9b2b6b4653a121856fe4574880, State: Initialized, Role: FOLLOWER
I20260929 20:23:01.969178 22996 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1aaadc9b2b6b4653a121856fe4574880 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1aaadc9b2b6b4653a121856fe4574880" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 41093 } }
I20260929 20:23:01.969383 22996 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1aaadc9b2b6b4653a121856fe4574880 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 20:23:01.969475 22996 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1aaadc9b2b6b4653a121856fe4574880 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 20:23:01.969599 22996 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1aaadc9b2b6b4653a121856fe4574880 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:23:01.971698 22996 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1aaadc9b2b6b4653a121856fe4574880 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1aaadc9b2b6b4653a121856fe4574880" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 41093 } }
I20260929 20:23:01.972136 22996 leader_election.cc:302] T 00000000000000000000000000000000 P 1aaadc9b2b6b4653a121856fe4574880 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1aaadc9b2b6b4653a121856fe4574880; no voters:
I20260929 20:23:01.972625 22996 leader_election.cc:288] T 00000000000000000000000000000000 P 1aaadc9b2b6b4653a121856fe4574880 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 20:23:01.972791 23001 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1aaadc9b2b6b4653a121856fe4574880 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:23:01.973485 23001 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1aaadc9b2b6b4653a121856fe4574880 [term 1 LEADER]: Becoming Leader. State: Replica: 1aaadc9b2b6b4653a121856fe4574880, State: Running, Role: LEADER
I20260929 20:23:01.974004 23001 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1aaadc9b2b6b4653a121856fe4574880 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1aaadc9b2b6b4653a121856fe4574880" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 41093 } }
I20260929 20:23:01.974670 22996 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1aaadc9b2b6b4653a121856fe4574880 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:23:01.976835 23003 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1aaadc9b2b6b4653a121856fe4574880 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1aaadc9b2b6b4653a121856fe4574880. Latest consensus state: current_term: 1 leader_uuid: "1aaadc9b2b6b4653a121856fe4574880" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1aaadc9b2b6b4653a121856fe4574880" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 41093 } } }
I20260929 20:23:01.977147 23003 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1aaadc9b2b6b4653a121856fe4574880 [sys.catalog]: This master's current role is: LEADER
I20260929 20:23:01.977625 23002 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1aaadc9b2b6b4653a121856fe4574880 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1aaadc9b2b6b4653a121856fe4574880" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1aaadc9b2b6b4653a121856fe4574880" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 41093 } } }
I20260929 20:23:01.977945 23002 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1aaadc9b2b6b4653a121856fe4574880 [sys.catalog]: This master's current role is: LEADER
I20260929 20:23:01.978111 23008 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:23:01.985189 23008 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:23:01.995005 23008 catalog_manager.cc:1347] Generated new cluster ID: d79831410b444884a56baef5d8456013
I20260929 20:23:01.995155 23008 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:23:02.028257 23008 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 20:23:02.029613 23008 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:23:02.035943 23008 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1aaadc9b2b6b4653a121856fe4574880: Generated new TSK 0
I20260929 20:23:02.036818 23008 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260929 20:23:02.078226 23026 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260929 20:23:02.078356 23026 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260929 20:23:02.079610 22999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:02.059804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57894 (local address 127.17.53.126:41093)
0929 20:23:02.059993 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0929 20:23:02.060003 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 20:23:02.060399 (+ 396us) server_negotiation.cc:407] Connection header received
0929 20:23:02.060658 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:23:02.060663 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:23:02.060731 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:23:02.060886 (+ 155us) server_negotiation.cc:226] Negotiated authn=JWT
0929 20:23:02.061382 (+ 496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:02.062120 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:02.063559 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:02.063994 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:02.064137 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:02.064688 (+ 551us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0929 20:23:02.078915 (+ 14227us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46427/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0929 20:23:02.079296 (+ 381us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:57894: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46427/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":58}
W20260929 20:23:02.079761 22999 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:57894: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46427/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260929 20:23:02.079875 23025 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:23:02.059718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:41093 (local address 127.0.0.1:57894)
0929 20:23:02.060292 (+ 574us) negotiation.cc:107] Waiting for socket to connect
0929 20:23:02.060311 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 20:23:02.060474 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:23:02.060903 (+ 429us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:23:02.060908 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:23:02.060920 (+ 12us) client_negotiation.cc:189] Negotiated authn=JWT
0929 20:23:02.061215 (+ 295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:23:02.061224 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:23:02.062269 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:23:02.062276 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:23:02.063415 (+ 1139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:23:02.063426 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:23:02.064187 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:23:02.064194 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:23:02.064347 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:02.064379 (+ 32us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0929 20:23:02.079163 (+ 14784us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46427/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0929 20:23:02.079429 (+ 266us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.17.53.126:41093: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:46427/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":466,"thread_start_us":321,"threads_started":1}
I20260929 20:23:02.081115 17621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu pid 22930
I20260929 20:23:02.082346 22991 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:23:02.177038 17621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu with pid 22930
2026-09-29T20:23:02Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (738 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260929 20:23:02.232735 17621 mini_oidc.cc:152] Starting JWKS server
I20260929 20:23:02.235064 17621 webserver.cc:460] Webserver started at https://127.0.0.1:37173/ using document root <none> and password file <none>
I20260929 20:23:02.235329 17621 mini_oidc.cc:189] Starting OIDC Discovery server
I20260929 20:23:02.235936 17621 webserver.cc:460] Webserver started at http://127.0.0.1:46619/ using document root <none> and password file <none>
2026-09-29T20:23:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T20:23:02Z Disabled control of system clock
I20260929 20:23:02.262171 17621 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
/tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790713339161492-17621-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790713339161492-17621-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790713339161492-17621-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790713339161492-17621-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.53.126:36879
--webserver_interface=127.17.53.126
--webserver_port=0
--builtin_ntp_servers=127.17.53.84:44835
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.53.126:36879
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37173/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260929 20:23:02.589583 23033 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:23:02.589855 23033 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:23:02.589937 23033 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:23:02.597189 23033 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:23:02.597316 23033 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:23:02.597353 23033 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:23:02.597402 23033 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:23:02.597446 23033 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260929 20:23:02.597499 23033 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260929 20:23:02.597538 23033 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:37173/jwks.json
I20260929 20:23:02.606734 23033 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.53.84:44835
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790713339161492-17621-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790713339161492-17621-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.53.126:36879
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.53.126:36879
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37173/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790713339161492-17621-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.53.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790713339161492-17621-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:23:02.608073 23033 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:23:02.609899 23033 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:23:02.622107 23039 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:23:02.622227 23038 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:23:02.622159 23041 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:23:02.622956 23033 server_base.cc:1034] running on GCE node
I20260929 20:23:02.623790 23033 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:23:02.625324 23033 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:23:02.626559 23033 hybrid_clock.cc:648] HybridClock initialized: now 1790713382626516 us; error 51 us; skew 500 ppm
I20260929 20:23:02.626977 23033 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 20:23:02.629293 23033 webserver.cc:460] Webserver started at http://127.17.53.126:38969/ using document root <none> and password file <none>
I20260929 20:23:02.630043 23033 fs_manager.cc:359] Metadata directory not provided
I20260929 20:23:02.630180 23033 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:23:02.630517 23033 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:23:02.633539 23033 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790713339161492-17621-0/minicluster-data/master-0/data/instance:
uuid: "5bd11acba1e14cccb268bd7cdaceb456"
format_stamp: "Formatted at 2026-09-29 20:23:02 on dist-test-slave-zbnc"
server_key: "18b4deb97bd03cd592b5add2f45098e1"
server_key_iv: "0b3c24eaaf431ccde39d205d2f8209bd"
server_key_version: "encryptionkey@0"
I20260929 20:23:02.634370 23033 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790713339161492-17621-0/minicluster-data/master-0/wal/instance:
uuid: "5bd11acba1e14cccb268bd7cdaceb456"
format_stamp: "Formatted at 2026-09-29 20:23:02 on dist-test-slave-zbnc"
server_key: "18b4deb97bd03cd592b5add2f45098e1"
server_key_iv: "0b3c24eaaf431ccde39d205d2f8209bd"
server_key_version: "encryptionkey@0"
I20260929 20:23:02.640228 23033 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260929 20:23:02.643841 23047 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:23:02.645298 23033 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 20:23:02.645480 23033 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790713339161492-17621-0/minicluster-data/master-0/data,/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790713339161492-17621-0/minicluster-data/master-0/wal
uuid: "5bd11acba1e14cccb268bd7cdaceb456"
format_stamp: "Formatted at 2026-09-29 20:23:02 on dist-test-slave-zbnc"
server_key: "18b4deb97bd03cd592b5add2f45098e1"
server_key_iv: "0b3c24eaaf431ccde39d205d2f8209bd"
server_key_version: "encryptionkey@0"
I20260929 20:23:02.645723 23033 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790713339161492-17621-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790713339161492-17621-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790713339161492-17621-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:23:02.667572 23033 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:23:02.668488 23033 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:23:02.668844 23033 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:23:02.687861 23033 rpc_server.cc:307] RPC server started. Bound to: 127.17.53.126:36879
I20260929 20:23:02.687893 23098 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.53.126:36879 every 8 connection(s)
I20260929 20:23:02.689224 23033 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790713339161492-17621-0/minicluster-data/master-0/data/info.pb
I20260929 20:23:02.690659 17621 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu as pid 23033
I20260929 20:23:02.690917 17621 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790713339161492-17621-0/minicluster-data/master-0/wal/instance
I20260929 20:23:02.691733 17621 external_mini_cluster.cc:1428] Setting key 329ef49351fa16ffb89f87f8de7ab2cb
I20260929 20:23:02.693830 23099 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:23:02.704896 23102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:02.693729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46640 (local address 127.17.53.126:36879)
0929 20:23:02.694680 (+ 951us) server_negotiation.cc:206] Beginning negotiation
0929 20:23:02.694699 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0929 20:23:02.694740 (+ 41us) server_negotiation.cc:407] Connection header received
0929 20:23:02.696317 (+ 1577us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:23:02.696357 (+ 40us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:23:02.696824 (+ 467us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:23:02.697241 (+ 417us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:23:02.697796 (+ 555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:02.699199 (+ 1403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:02.700509 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:02.701160 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:02.701354 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:02.702581 (+ 1227us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:23:02.702591 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:23:02.702604 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:23:02.703231 (+ 627us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:23:02.703309 (+ 78us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:23:02.703332 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:23:02.703456 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0929 20:23:02.703679 (+ 223us) server_negotiation.cc:299] Negotiation successful
0929 20:23:02.703994 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":550,"thread_start_us":325,"threads_started":1}
I20260929 20:23:02.704994 23099 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5bd11acba1e14cccb268bd7cdaceb456: Bootstrap starting.
I20260929 20:23:02.708282 23099 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5bd11acba1e14cccb268bd7cdaceb456: Neither blocks nor log segments found. Creating new log.
I20260929 20:23:02.708698 17621 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 20:23:02.709342 23099 log.cc:824] T 00000000000000000000000000000000 P 5bd11acba1e14cccb268bd7cdaceb456: Log is configured to *not* fsync() on all Append() calls
I20260929 20:23:02.712364 23099 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5bd11acba1e14cccb268bd7cdaceb456: No bootstrap required, opened a new log
I20260929 20:23:02.719731 23099 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5bd11acba1e14cccb268bd7cdaceb456 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bd11acba1e14cccb268bd7cdaceb456" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 36879 } }
I20260929 20:23:02.720242 23099 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5bd11acba1e14cccb268bd7cdaceb456 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:23:02.720351 23099 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5bd11acba1e14cccb268bd7cdaceb456 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5bd11acba1e14cccb268bd7cdaceb456, State: Initialized, Role: FOLLOWER
I20260929 20:23:02.720927 23099 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5bd11acba1e14cccb268bd7cdaceb456 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bd11acba1e14cccb268bd7cdaceb456" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 36879 } }
I20260929 20:23:02.721175 23099 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5bd11acba1e14cccb268bd7cdaceb456 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 20:23:02.721328 23099 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5bd11acba1e14cccb268bd7cdaceb456 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 20:23:02.721504 23099 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5bd11acba1e14cccb268bd7cdaceb456 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:23:02.723724 23099 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5bd11acba1e14cccb268bd7cdaceb456 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bd11acba1e14cccb268bd7cdaceb456" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 36879 } }
I20260929 20:23:02.724244 23099 leader_election.cc:302] T 00000000000000000000000000000000 P 5bd11acba1e14cccb268bd7cdaceb456 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5bd11acba1e14cccb268bd7cdaceb456; no voters:
I20260929 20:23:02.724849 23099 leader_election.cc:288] T 00000000000000000000000000000000 P 5bd11acba1e14cccb268bd7cdaceb456 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 20:23:02.725040 23104 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5bd11acba1e14cccb268bd7cdaceb456 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:23:02.726119 23104 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5bd11acba1e14cccb268bd7cdaceb456 [term 1 LEADER]: Becoming Leader. State: Replica: 5bd11acba1e14cccb268bd7cdaceb456, State: Running, Role: LEADER
I20260929 20:23:02.726658 23104 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5bd11acba1e14cccb268bd7cdaceb456 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bd11acba1e14cccb268bd7cdaceb456" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 36879 } }
I20260929 20:23:02.727102 23099 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5bd11acba1e14cccb268bd7cdaceb456 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:23:02.729702 23106 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5bd11acba1e14cccb268bd7cdaceb456 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5bd11acba1e14cccb268bd7cdaceb456. Latest consensus state: current_term: 1 leader_uuid: "5bd11acba1e14cccb268bd7cdaceb456" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bd11acba1e14cccb268bd7cdaceb456" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 36879 } } }
I20260929 20:23:02.729667 23105 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5bd11acba1e14cccb268bd7cdaceb456 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5bd11acba1e14cccb268bd7cdaceb456" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bd11acba1e14cccb268bd7cdaceb456" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 36879 } } }
I20260929 20:23:02.730216 23105 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5bd11acba1e14cccb268bd7cdaceb456 [sys.catalog]: This master's current role is: LEADER
I20260929 20:23:02.731120 23113 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:23:02.730214 23106 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5bd11acba1e14cccb268bd7cdaceb456 [sys.catalog]: This master's current role is: LEADER
I20260929 20:23:02.737082 23113 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:23:02.748685 23113 catalog_manager.cc:1347] Generated new cluster ID: d5a8122230de43dd98d312d5d12c450e
I20260929 20:23:02.748826 23113 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:23:02.758773 23113 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 20:23:02.760530 23113 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:23:02.769349 23113 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5bd11acba1e14cccb268bd7cdaceb456: Generated new TSK 0
I20260929 20:23:02.770184 23113 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 20:23:02.803326 23102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:02.793946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46644 (local address 127.17.53.126:36879)
0929 20:23:02.794169 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0929 20:23:02.794177 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 20:23:02.794669 (+ 492us) server_negotiation.cc:407] Connection header received
0929 20:23:02.794963 (+ 294us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:23:02.794967 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:23:02.795037 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:23:02.795165 (+ 128us) server_negotiation.cc:226] Negotiated authn=JWT
0929 20:23:02.795741 (+ 576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:02.796465 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:02.798497 (+ 2032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:02.798989 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:02.799080 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:02.799710 (+ 630us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0929 20:23:02.802441 (+ 2731us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37173/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0929 20:23:02.802901 (+ 460us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:46644: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37173/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":68}
W20260929 20:23:02.803483 23102 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:46644: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37173/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260929 20:23:02.803555 23128 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:23:02.793922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:36879 (local address 127.0.0.1:46644)
0929 20:23:02.794559 (+ 637us) negotiation.cc:107] Waiting for socket to connect
0929 20:23:02.794580 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 20:23:02.794732 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:23:02.795221 (+ 489us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:23:02.795226 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:23:02.795240 (+ 14us) client_negotiation.cc:189] Negotiated authn=JWT
0929 20:23:02.795581 (+ 341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:23:02.795594 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:23:02.796622 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:23:02.796628 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:23:02.798360 (+ 1732us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:23:02.798376 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:23:02.799185 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:23:02.799191 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:23:02.799405 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:02.799454 (+ 49us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0929 20:23:02.802719 (+ 3265us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37173/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0929 20:23:02.802988 (+ 269us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.17.53.126:36879: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37173/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":493,"thread_start_us":342,"threads_started":1}
I20260929 20:23:02.804970 17621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu pid 23033
I20260929 20:23:02.806171 23094 generic_service.cc:196] Checking for leaks (request via RPC)
E20260929 20:23:02.817248 23129 webserver.cc:533] Webserver: sslize failed:
E20260929 20:23:02.817391 23129 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260929 20:23:02.895370 17621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu with pid 23033
2026-09-29T20:23:02Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (714 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-29T20:23:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T20:23:02Z Disabled control of system clock
I20260929 20:23:02.974498 17621 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
/tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790713339161492-17621-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790713339161492-17621-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790713339161492-17621-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790713339161492-17621-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.53.126:32979
--webserver_interface=127.17.53.126
--webserver_port=0
--builtin_ntp_servers=127.17.53.84:40291
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.53.126:32979
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790713339161492-17621-0/testchainca.cert with env {}
W20260929 20:23:03.308009 23134 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:23:03.308348 23134 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:23:03.308439 23134 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:23:03.315590 23134 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:23:03.315701 23134 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:23:03.315739 23134 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:23:03.315789 23134 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:23:03.315833 23134 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260929 20:23:03.315907 23134 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260929 20:23:03.315953 23134 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260929 20:23:03.326047 23134 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.53.84:40291
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790713339161492-17621-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790713339161492-17621-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.53.126:32979
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.53.126:32979
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790713339161492-17621-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.53.126
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790713339161492-17621-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790713339161492-17621-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:23:03.327354 23134 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:23:03.328706 23134 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:23:03.340873 23139 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:23:03.340906 23140 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:23:03.340958 23142 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:23:03.342276 23134 server_base.cc:1034] running on GCE node
I20260929 20:23:03.343122 23134 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:23:03.344668 23134 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:23:03.345867 23134 hybrid_clock.cc:648] HybridClock initialized: now 1790713383345806 us; error 54 us; skew 500 ppm
I20260929 20:23:03.346220 23134 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 20:23:03.348726 23134 webserver.cc:460] Webserver started at http://127.17.53.126:43413/ using document root <none> and password file <none>
I20260929 20:23:03.349474 23134 fs_manager.cc:359] Metadata directory not provided
I20260929 20:23:03.349587 23134 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:23:03.349923 23134 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:23:03.352952 23134 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790713339161492-17621-0/minicluster-data/master-0/data/instance:
uuid: "2f4f81c001294d94a96669b8aca1869e"
format_stamp: "Formatted at 2026-09-29 20:23:03 on dist-test-slave-zbnc"
server_key: "34b575b6f4012ff355a704376066a429"
server_key_iv: "1be65abced7cfd12e9dbcf2ba651ce76"
server_key_version: "encryptionkey@0"
I20260929 20:23:03.353787 23134 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790713339161492-17621-0/minicluster-data/master-0/wal/instance:
uuid: "2f4f81c001294d94a96669b8aca1869e"
format_stamp: "Formatted at 2026-09-29 20:23:03 on dist-test-slave-zbnc"
server_key: "34b575b6f4012ff355a704376066a429"
server_key_iv: "1be65abced7cfd12e9dbcf2ba651ce76"
server_key_version: "encryptionkey@0"
I20260929 20:23:03.359071 23134 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.002s
I20260929 20:23:03.362421 23148 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:23:03.363889 23134 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 20:23:03.364071 23134 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790713339161492-17621-0/minicluster-data/master-0/data,/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790713339161492-17621-0/minicluster-data/master-0/wal
uuid: "2f4f81c001294d94a96669b8aca1869e"
format_stamp: "Formatted at 2026-09-29 20:23:03 on dist-test-slave-zbnc"
server_key: "34b575b6f4012ff355a704376066a429"
server_key_iv: "1be65abced7cfd12e9dbcf2ba651ce76"
server_key_version: "encryptionkey@0"
I20260929 20:23:03.364326 23134 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790713339161492-17621-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790713339161492-17621-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790713339161492-17621-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:23:03.388634 23134 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:23:03.389593 23134 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:23:03.389894 23134 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:23:03.409659 23134 rpc_server.cc:307] RPC server started. Bound to: 127.17.53.126:32979
I20260929 20:23:03.409754 23199 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.53.126:32979 every 8 connection(s)
I20260929 20:23:03.411208 23134 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790713339161492-17621-0/minicluster-data/master-0/data/info.pb
I20260929 20:23:03.415370 23200 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:23:03.420480 17621 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu as pid 23134
I20260929 20:23:03.420640 17621 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790713339161492-17621-0/minicluster-data/master-0/wal/instance
I20260929 20:23:03.421322 17621 external_mini_cluster.cc:1428] Setting key 1e9f5f9cde2b05d97f8d2e1d4a4c8e03
I20260929 20:23:03.426643 23200 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2f4f81c001294d94a96669b8aca1869e: Bootstrap starting.
I20260929 20:23:03.430464 23200 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2f4f81c001294d94a96669b8aca1869e: Neither blocks nor log segments found. Creating new log.
I20260929 20:23:03.431710 23200 log.cc:824] T 00000000000000000000000000000000 P 2f4f81c001294d94a96669b8aca1869e: Log is configured to *not* fsync() on all Append() calls
I20260929 20:23:03.434702 23200 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2f4f81c001294d94a96669b8aca1869e: No bootstrap required, opened a new log
I20260929 20:23:03.435752 23203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:03.423539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44080 (local address 127.17.53.126:32979)
0929 20:23:03.424684 (+ 1145us) server_negotiation.cc:206] Beginning negotiation
0929 20:23:03.424695 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 20:23:03.424742 (+ 47us) server_negotiation.cc:407] Connection header received
0929 20:23:03.426353 (+ 1611us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:23:03.426378 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:23:03.426931 (+ 553us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:23:03.427306 (+ 375us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:23:03.427976 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:03.429286 (+ 1310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:03.431167 (+ 1881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:03.431933 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:03.432063 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:03.433399 (+ 1336us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:23:03.433407 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:23:03.433422 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:23:03.433967 (+ 545us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:23:03.434043 (+ 76us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:23:03.434064 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:23:03.434207 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0929 20:23:03.434431 (+ 224us) server_negotiation.cc:299] Negotiation successful
0929 20:23:03.434751 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":639,"thread_start_us":400,"threads_started":1}
I20260929 20:23:03.439808 17621 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 20:23:03.443599 23200 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2f4f81c001294d94a96669b8aca1869e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f4f81c001294d94a96669b8aca1869e" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 32979 } }
I20260929 20:23:03.443986 23200 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2f4f81c001294d94a96669b8aca1869e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:23:03.444099 23200 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2f4f81c001294d94a96669b8aca1869e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f4f81c001294d94a96669b8aca1869e, State: Initialized, Role: FOLLOWER
I20260929 20:23:03.444638 23200 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2f4f81c001294d94a96669b8aca1869e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f4f81c001294d94a96669b8aca1869e" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 32979 } }
I20260929 20:23:03.444890 23200 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2f4f81c001294d94a96669b8aca1869e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 20:23:03.445024 23200 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2f4f81c001294d94a96669b8aca1869e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 20:23:03.445204 23200 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2f4f81c001294d94a96669b8aca1869e [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:23:03.447577 23200 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2f4f81c001294d94a96669b8aca1869e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f4f81c001294d94a96669b8aca1869e" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 32979 } }
I20260929 20:23:03.448076 23200 leader_election.cc:302] T 00000000000000000000000000000000 P 2f4f81c001294d94a96669b8aca1869e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2f4f81c001294d94a96669b8aca1869e; no voters:
I20260929 20:23:03.448748 23200 leader_election.cc:288] T 00000000000000000000000000000000 P 2f4f81c001294d94a96669b8aca1869e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 20:23:03.448909 23205 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2f4f81c001294d94a96669b8aca1869e [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:23:03.449949 23205 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2f4f81c001294d94a96669b8aca1869e [term 1 LEADER]: Becoming Leader. State: Replica: 2f4f81c001294d94a96669b8aca1869e, State: Running, Role: LEADER
I20260929 20:23:03.450515 23205 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2f4f81c001294d94a96669b8aca1869e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f4f81c001294d94a96669b8aca1869e" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 32979 } }
I20260929 20:23:03.451185 23200 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2f4f81c001294d94a96669b8aca1869e [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:23:03.453397 23207 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f4f81c001294d94a96669b8aca1869e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2f4f81c001294d94a96669b8aca1869e. Latest consensus state: current_term: 1 leader_uuid: "2f4f81c001294d94a96669b8aca1869e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f4f81c001294d94a96669b8aca1869e" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 32979 } } }
I20260929 20:23:03.453691 23207 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f4f81c001294d94a96669b8aca1869e [sys.catalog]: This master's current role is: LEADER
I20260929 20:23:03.454592 23211 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:23:03.454377 23206 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f4f81c001294d94a96669b8aca1869e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2f4f81c001294d94a96669b8aca1869e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2f4f81c001294d94a96669b8aca1869e" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 32979 } } }
I20260929 20:23:03.454754 23206 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f4f81c001294d94a96669b8aca1869e [sys.catalog]: This master's current role is: LEADER
I20260929 20:23:03.460150 23211 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:23:03.469267 23211 catalog_manager.cc:1347] Generated new cluster ID: 0d5cd89751934a8481bcdc965d7b1157
I20260929 20:23:03.469379 23211 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:23:03.483808 23211 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 20:23:03.485751 23211 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:23:03.495576 23211 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2f4f81c001294d94a96669b8aca1869e: Generated new TSK 0
I20260929 20:23:03.496693 23211 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 20:23:03.591578 23203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:03.515940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44090 (local address 127.17.53.126:32979)
0929 20:23:03.516146 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0929 20:23:03.516152 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:23:03.516633 (+ 481us) server_negotiation.cc:407] Connection header received
0929 20:23:03.516839 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:23:03.516844 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:23:03.516944 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:23:03.517112 (+ 168us) server_negotiation.cc:226] Negotiated authn=JWT
0929 20:23:03.517608 (+ 496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:03.518780 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:03.520274 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:03.520973 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:03.521100 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:03.521636 (+ 536us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0929 20:23:03.590778 (+ 69142us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0929 20:23:03.591236 (+ 458us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:44090: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":65}
W20260929 20:23:03.591722 23203 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:44090: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260929 20:23:03.591735 23229 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:23:03.515904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:32979 (local address 127.0.0.1:44090)
0929 20:23:03.516534 (+ 630us) negotiation.cc:107] Waiting for socket to connect
0929 20:23:03.516551 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 20:23:03.516672 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:23:03.517192 (+ 520us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:23:03.517196 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:23:03.517207 (+ 11us) client_negotiation.cc:189] Negotiated authn=JWT
0929 20:23:03.517452 (+ 245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:23:03.517459 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:23:03.518993 (+ 1534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:23:03.518998 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:23:03.520119 (+ 1121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:23:03.520130 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:23:03.521182 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:23:03.521188 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:23:03.521331 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:03.521367 (+ 36us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0929 20:23:03.591155 (+ 69788us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0929 20:23:03.591382 (+ 227us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.17.53.126:32979: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":498,"thread_start_us":341,"threads_started":1}
I20260929 20:23:03.593437 17621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu pid 23134
I20260929 20:23:03.594733 23195 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:23:03.692279 17621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu with pid 23134
2026-09-29T20:23:03Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (779 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (412 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (415 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790713339161492-17621-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 20:23:04 dist-test-slave-zbnc krb5kdc[23244](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 20:23:04 dist-test-slave-zbnc krb5kdc[23244](info): set up 2 sockets
Sep 29 20:23:04 dist-test-slave-zbnc krb5kdc[23244](info): commencing operation
krb5kdc: starting...
W20260929 20:23:05.612939 17621 mini_kdc.cc:121] Time spent starting KDC: real 1.056s user 0.001s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790713339161492-17621-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790713339161492-17621-0/krb5kdc/test-user.keytab.
Sep 29 20:23:05 dist-test-slave-zbnc krb5kdc[23244](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713385, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T20:23:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T20:23:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.17.53.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.53.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.53.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.126.keytab.
Entry for principal kudu/127.17.53.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.126.keytab.
I20260929 20:23:05.827159 17621 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
/tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790713339161492-17621-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790713339161492-17621-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790713339161492-17621-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790713339161492-17621-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.53.126:41283
--webserver_interface=127.17.53.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.126.keytab
--principal=kudu/127.17.53.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.53.84:39137
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.53.126:41283
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790713339161492-17621-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790713339161492-17621-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790713339161492-17621-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:23:06.177686 23257 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:23:06.178037 23257 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:23:06.178211 23257 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:23:06.185619 23257 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:23:06.185793 23257 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:23:06.185907 23257 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:23:06.186005 23257 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:23:06.186101 23257 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:23:06.196456 23257 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.53.84:39137
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790713339161492-17621-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790713339161492-17621-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.53.126:41283
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.126.keytab
--principal=kudu/127.17.53.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.53.126:41283
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790713339161492-17621-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.53.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790713339161492-17621-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:23:06.197994 23257 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:23:06.199424 23257 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:23:06.211731 23262 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:23:06.211748 23265 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:23:06.211678 23263 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:23:06.212652 23257 server_base.cc:1034] running on GCE node
I20260929 20:23:06.213512 23257 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:23:06.215082 23257 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:23:06.216324 23257 hybrid_clock.cc:648] HybridClock initialized: now 1790713386216283 us; error 53 us; skew 500 ppm
I20260929 20:23:06.216796 23257 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:23:06 dist-test-slave-zbnc krb5kdc[23244](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713386, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.53.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:23:06.225440 23257 init.cc:377] Logged in from keytab as kudu/127.17.53.126@KRBTEST.COM (short username kudu)
I20260929 20:23:06.227197 23257 webserver.cc:460] Webserver started at http://127.17.53.126:39265/ using document root <none> and password file <none>
I20260929 20:23:06.227933 23257 fs_manager.cc:359] Metadata directory not provided
I20260929 20:23:06.228084 23257 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:23:06.228412 23257 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:23:06.231534 23257 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790713339161492-17621-0/minicluster-data/master-0/data/instance:
uuid: "c768bc9fe0d44a609c3ff45a2653c848"
format_stamp: "Formatted at 2026-09-29 20:23:06 on dist-test-slave-zbnc"
server_key: "8f35dca7b4c0713f70ba7d275c92304c"
server_key_iv: "e82f508c2f1ba70a4c0d635a45e94016"
server_key_version: "encryptionkey@0"
I20260929 20:23:06.232328 23257 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790713339161492-17621-0/minicluster-data/master-0/wal/instance:
uuid: "c768bc9fe0d44a609c3ff45a2653c848"
format_stamp: "Formatted at 2026-09-29 20:23:06 on dist-test-slave-zbnc"
server_key: "8f35dca7b4c0713f70ba7d275c92304c"
server_key_iv: "e82f508c2f1ba70a4c0d635a45e94016"
server_key_version: "encryptionkey@0"
I20260929 20:23:06.237711 23257 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 20:23:06.241341 23272 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:23:06.242785 23257 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 20:23:06.242995 23257 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790713339161492-17621-0/minicluster-data/master-0/data,/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790713339161492-17621-0/minicluster-data/master-0/wal
uuid: "c768bc9fe0d44a609c3ff45a2653c848"
format_stamp: "Formatted at 2026-09-29 20:23:06 on dist-test-slave-zbnc"
server_key: "8f35dca7b4c0713f70ba7d275c92304c"
server_key_iv: "e82f508c2f1ba70a4c0d635a45e94016"
server_key_version: "encryptionkey@0"
I20260929 20:23:06.243201 23257 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790713339161492-17621-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790713339161492-17621-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790713339161492-17621-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:23:06.266284 23257 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:23:06.270411 23257 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:23:06.270716 23257 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:23:06.289968 23257 rpc_server.cc:307] RPC server started. Bound to: 127.17.53.126:41283
I20260929 20:23:06.290001 23323 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.53.126:41283 every 8 connection(s)
I20260929 20:23:06.291421 23257 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790713339161492-17621-0/minicluster-data/master-0/data/info.pb
I20260929 20:23:06.295181 23324 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:23:06.299418 17621 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu as pid 23257
I20260929 20:23:06.299641 17621 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790713339161492-17621-0/minicluster-data/master-0/wal/instance
I20260929 20:23:06.300285 17621 external_mini_cluster.cc:1428] Setting key a51ff68d9eea5b155a90570d76b81a66
I20260929 20:23:06.307401 23324 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c768bc9fe0d44a609c3ff45a2653c848: Bootstrap starting.
I20260929 20:23:06.311221 23324 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c768bc9fe0d44a609c3ff45a2653c848: Neither blocks nor log segments found. Creating new log.
I20260929 20:23:06.312444 23324 log.cc:824] T 00000000000000000000000000000000 P c768bc9fe0d44a609c3ff45a2653c848: Log is configured to *not* fsync() on all Append() calls
I20260929 20:23:06.315719 23324 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c768bc9fe0d44a609c3ff45a2653c848: No bootstrap required, opened a new log
Sep 29 20:23:06 dist-test-slave-zbnc krb5kdc[23244](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713385, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.53.126@KRBTEST.COM
I20260929 20:23:06.325719 23327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:06.302333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59980 (local address 127.17.53.126:41283)
0929 20:23:06.303164 (+ 831us) server_negotiation.cc:206] Beginning negotiation
0929 20:23:06.303186 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0929 20:23:06.303236 (+ 50us) server_negotiation.cc:407] Connection header received
0929 20:23:06.305775 (+ 2539us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:23:06.305805 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:23:06.306313 (+ 508us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:23:06.306788 (+ 475us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:23:06.308240 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:06.310303 (+ 2063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:06.311799 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:06.312859 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:06.313108 (+ 249us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:06.317858 (+ 4750us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:23:06.317866 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:23:06.317880 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:23:06.317917 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:23:06.321232 (+ 3315us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:23:06.322155 (+ 923us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:23:06.322161 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:23:06.322167 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:23:06.322353 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:23:06.322826 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:23:06.322830 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:23:06.322832 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:23:06.323471 (+ 639us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:23:06.323702 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0929 20:23:06.324209 (+ 507us) server_negotiation.cc:299] Negotiation successful
0929 20:23:06.324617 (+ 408us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":575,"thread_start_us":332,"threads_started":1}
I20260929 20:23:06.326332 23324 raft_consensus.cc:348] T 00000000000000000000000000000000 P c768bc9fe0d44a609c3ff45a2653c848 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c768bc9fe0d44a609c3ff45a2653c848" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 41283 } }
I20260929 20:23:06.326764 23324 raft_consensus.cc:374] T 00000000000000000000000000000000 P c768bc9fe0d44a609c3ff45a2653c848 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:23:06.326917 23324 raft_consensus.cc:729] T 00000000000000000000000000000000 P c768bc9fe0d44a609c3ff45a2653c848 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c768bc9fe0d44a609c3ff45a2653c848, State: Initialized, Role: FOLLOWER
I20260929 20:23:06.327535 23324 consensus_queue.cc:260] T 00000000000000000000000000000000 P c768bc9fe0d44a609c3ff45a2653c848 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c768bc9fe0d44a609c3ff45a2653c848" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 41283 } }
I20260929 20:23:06.327814 23324 raft_consensus.cc:388] T 00000000000000000000000000000000 P c768bc9fe0d44a609c3ff45a2653c848 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 20:23:06.327981 23324 raft_consensus.cc:482] T 00000000000000000000000000000000 P c768bc9fe0d44a609c3ff45a2653c848 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 20:23:06.328190 23324 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c768bc9fe0d44a609c3ff45a2653c848 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:23:06.331117 23324 raft_consensus.cc:504] T 00000000000000000000000000000000 P c768bc9fe0d44a609c3ff45a2653c848 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c768bc9fe0d44a609c3ff45a2653c848" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 41283 } }
I20260929 20:23:06.331619 23324 leader_election.cc:302] T 00000000000000000000000000000000 P c768bc9fe0d44a609c3ff45a2653c848 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c768bc9fe0d44a609c3ff45a2653c848; no voters:
I20260929 20:23:06.332222 23324 leader_election.cc:288] T 00000000000000000000000000000000 P c768bc9fe0d44a609c3ff45a2653c848 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 20:23:06.332381 23329 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c768bc9fe0d44a609c3ff45a2653c848 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:23:06.333132 23329 raft_consensus.cc:686] T 00000000000000000000000000000000 P c768bc9fe0d44a609c3ff45a2653c848 [term 1 LEADER]: Becoming Leader. State: Replica: c768bc9fe0d44a609c3ff45a2653c848, State: Running, Role: LEADER
I20260929 20:23:06.333770 23329 consensus_queue.cc:237] T 00000000000000000000000000000000 P c768bc9fe0d44a609c3ff45a2653c848 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c768bc9fe0d44a609c3ff45a2653c848" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 41283 } }
I20260929 20:23:06.334441 23324 sys_catalog.cc:564] T 00000000000000000000000000000000 P c768bc9fe0d44a609c3ff45a2653c848 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:23:06.336958 23330 sys_catalog.cc:455] T 00000000000000000000000000000000 P c768bc9fe0d44a609c3ff45a2653c848 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c768bc9fe0d44a609c3ff45a2653c848" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c768bc9fe0d44a609c3ff45a2653c848" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 41283 } } }
I20260929 20:23:06.337471 23330 sys_catalog.cc:458] T 00000000000000000000000000000000 P c768bc9fe0d44a609c3ff45a2653c848 [sys.catalog]: This master's current role is: LEADER
I20260929 20:23:06.337965 23331 sys_catalog.cc:455] T 00000000000000000000000000000000 P c768bc9fe0d44a609c3ff45a2653c848 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c768bc9fe0d44a609c3ff45a2653c848. Latest consensus state: current_term: 1 leader_uuid: "c768bc9fe0d44a609c3ff45a2653c848" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c768bc9fe0d44a609c3ff45a2653c848" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 41283 } } }
I20260929 20:23:06.338308 23331 sys_catalog.cc:458] T 00000000000000000000000000000000 P c768bc9fe0d44a609c3ff45a2653c848 [sys.catalog]: This master's current role is: LEADER
I20260929 20:23:06.340011 23338 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:23:06.345562 23338 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:23:06.354722 23338 catalog_manager.cc:1347] Generated new cluster ID: db6c5c7421e848338ced136dab242257
I20260929 20:23:06.354898 23338 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:23:06.376775 23338 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 20:23:06.378721 23338 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:23:06.387166 23338 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c768bc9fe0d44a609c3ff45a2653c848: Generated new TSK 0
I20260929 20:23:06.388377 23338 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.17.53.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.53.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.53.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.65.keytab.
Entry for principal kudu/127.17.53.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.65.keytab.
I20260929 20:23:06.443361 17621 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
/tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790713339161492-17621-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790713339161492-17621-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790713339161492-17621-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790713339161492-17621-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.53.65:0
--local_ip_for_outbound_sockets=127.17.53.65
--webserver_interface=127.17.53.65
--webserver_port=0
--tserver_master_addrs=127.17.53.126:41283
--keytab_file=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.65.keytab
--principal=kudu/127.17.53.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.53.84:39137
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790713339161492-17621-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790713339161492-17621-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790713339161492-17621-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:23:06.784500 23350 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:23:06.784789 23350 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:23:06.784884 23350 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:23:06.792335 23350 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:23:06.792500 23350 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:23:06.792701 23350 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.53.65
I20260929 20:23:06.802697 23350 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.53.84:39137
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790713339161492-17621-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790713339161492-17621-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.65.keytab
--principal=kudu/127.17.53.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.53.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790713339161492-17621-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.17.53.65
--webserver_port=0
--tserver_master_addrs=127.17.53.126:41283
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.53.65
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790713339161492-17621-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:23:06.804057 23350 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:23:06.805374 23350 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:23:06.819387 23358 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:23:06.819492 23356 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:23:06.819442 23355 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:23:06.820292 23350 server_base.cc:1034] running on GCE node
I20260929 20:23:06.820916 23350 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:23:06.821918 23350 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:23:06.823118 23350 hybrid_clock.cc:648] HybridClock initialized: now 1790713386823053 us; error 71 us; skew 500 ppm
I20260929 20:23:06.823462 23350 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:23:06 dist-test-slave-zbnc krb5kdc[23244](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713386, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.53.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:23:06.827288 23350 init.cc:377] Logged in from keytab as kudu/127.17.53.65@KRBTEST.COM (short username kudu)
I20260929 20:23:06.828840 23350 webserver.cc:460] Webserver started at http://127.17.53.65:33331/ using document root <none> and password file <none>
I20260929 20:23:06.829541 23350 fs_manager.cc:359] Metadata directory not provided
I20260929 20:23:06.829675 23350 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:23:06.829994 23350 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:23:06.833038 23350 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790713339161492-17621-0/minicluster-data/ts-0/data/instance:
uuid: "015a9db984a94ef5820f61b31658e9d2"
format_stamp: "Formatted at 2026-09-29 20:23:06 on dist-test-slave-zbnc"
server_key: "912322569cdcfd46281eada11c329940"
server_key_iv: "8562f4c763d15e35ea699943aa74f0fd"
server_key_version: "encryptionkey@0"
I20260929 20:23:06.833846 23350 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790713339161492-17621-0/minicluster-data/ts-0/wal/instance:
uuid: "015a9db984a94ef5820f61b31658e9d2"
format_stamp: "Formatted at 2026-09-29 20:23:06 on dist-test-slave-zbnc"
server_key: "912322569cdcfd46281eada11c329940"
server_key_iv: "8562f4c763d15e35ea699943aa74f0fd"
server_key_version: "encryptionkey@0"
I20260929 20:23:06.839741 23350 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260929 20:23:06.843276 23365 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:23:06.844744 23350 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 20:23:06.844914 23350 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790713339161492-17621-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790713339161492-17621-0/minicluster-data/ts-0/wal
uuid: "015a9db984a94ef5820f61b31658e9d2"
format_stamp: "Formatted at 2026-09-29 20:23:06 on dist-test-slave-zbnc"
server_key: "912322569cdcfd46281eada11c329940"
server_key_iv: "8562f4c763d15e35ea699943aa74f0fd"
server_key_version: "encryptionkey@0"
I20260929 20:23:06.845106 23350 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790713339161492-17621-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790713339161492-17621-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790713339161492-17621-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:23:06.862012 23350 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:23:06.866205 23350 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:23:06.866495 23350 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:23:06.867524 23350 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:23:06.868860 23350 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:23:06.868932 23350 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:23:06.869099 23350 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:23:06.869158 23350 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:23:06.903848 23350 rpc_server.cc:307] RPC server started. Bound to: 127.17.53.65:43749
I20260929 20:23:06.903885 23477 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.53.65:43749 every 8 connection(s)
I20260929 20:23:06.905339 23350 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790713339161492-17621-0/minicluster-data/ts-0/data/info.pb
I20260929 20:23:06.914772 17621 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu as pid 23350
I20260929 20:23:06.915017 17621 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790713339161492-17621-0/minicluster-data/ts-0/wal/instance
I20260929 20:23:06.915668 17621 external_mini_cluster.cc:1428] Setting key bb09087cb6f6d76c0234878b3618b36a
Sep 29 20:23:06 dist-test-slave-zbnc krb5kdc[23244](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713386, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.53.65@KRBTEST.COM for kudu/127.17.53.126@KRBTEST.COM
I20260929 20:23:06.926374 23480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:06.907963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.53.65:40729 (local address 127.17.53.126:41283)
0929 20:23:06.908418 (+ 455us) server_negotiation.cc:206] Beginning negotiation
0929 20:23:06.908423 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:23:06.909285 (+ 862us) server_negotiation.cc:407] Connection header received
0929 20:23:06.911132 (+ 1847us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:23:06.911137 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:23:06.911239 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:23:06.911360 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:23:06.912884 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:06.913618 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:06.915357 (+ 1739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:06.915796 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:06.915947 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:06.920108 (+ 4161us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:23:06.920113 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:23:06.920116 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:23:06.920147 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:23:06.922671 (+ 2524us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:23:06.923715 (+ 1044us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:23:06.923718 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:23:06.923721 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:23:06.923838 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:23:06.924314 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:23:06.924321 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:23:06.924325 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:23:06.924715 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:23:06.924867 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0929 20:23:06.925622 (+ 755us) server_negotiation.cc:299] Negotiation successful
0929 20:23:06.925880 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":362,"thread_start_us":256,"threads_started":1}
I20260929 20:23:06.927142 23481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:06.908401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:41283 (local address 127.17.53.65:40729)
0929 20:23:06.909114 (+ 713us) negotiation.cc:107] Waiting for socket to connect
0929 20:23:06.909153 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0929 20:23:06.910797 (+ 1644us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:23:06.911611 (+ 814us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:23:06.911620 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:23:06.912133 (+ 513us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:23:06.912711 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:23:06.912728 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:23:06.913791 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:23:06.913797 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:23:06.915218 (+ 1421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:23:06.915230 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:23:06.916036 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:23:06.916044 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:23:06.916305 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:06.916996 (+ 691us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:23:06.917026 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:23:06.919890 (+ 2864us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:23:06.922830 (+ 2940us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:23:06.922842 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:23:06.922888 (+ 46us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:23:06.923562 (+ 674us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:23:06.923992 (+ 430us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:23:06.923996 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:23:06.923998 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:23:06.924193 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:23:06.924848 (+ 655us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:23:06.924854 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:23:06.925224 (+ 370us) client_negotiation.cc:769] Sending connection context
0929 20:23:06.925458 (+ 234us) client_negotiation.cc:240] Negotiation successful
0929 20:23:06.925881 (+ 423us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":478,"thread_start_us":286,"threads_started":1}
I20260929 20:23:06.929006 23478 heartbeater.cc:347] Connected to a master server at 127.17.53.126:41283
I20260929 20:23:06.929390 23478 heartbeater.cc:464] Registering TS with master...
I20260929 20:23:06.930256 23478 heartbeater.cc:511] Master 127.17.53.126:41283 requested a full tablet report, sending...
I20260929 20:23:06.932870 23289 ts_manager.cc:194] Registered new tserver with Master: 015a9db984a94ef5820f61b31658e9d2 (127.17.53.65:43749)
I20260929 20:23:06.934664 23289 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.53.65@KRBTEST.COM'} at 127.17.53.65:40729
WARNING: no policy specified for kudu/127.17.53.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.53.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.53.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.66.keytab.
Entry for principal kudu/127.17.53.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.66.keytab.
I20260929 20:23:06.966102 17621 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
/tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790713339161492-17621-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790713339161492-17621-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790713339161492-17621-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790713339161492-17621-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.53.66:0
--local_ip_for_outbound_sockets=127.17.53.66
--webserver_interface=127.17.53.66
--webserver_port=0
--tserver_master_addrs=127.17.53.126:41283
--keytab_file=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.66.keytab
--principal=kudu/127.17.53.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.53.84:39137
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790713339161492-17621-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790713339161492-17621-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790713339161492-17621-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:23:07.298681 23484 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:23:07.299072 23484 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:23:07.299245 23484 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:23:07.306780 23484 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:23:07.306967 23484 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:23:07.307158 23484 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.53.66
I20260929 20:23:07.316938 23484 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.53.84:39137
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790713339161492-17621-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790713339161492-17621-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.66.keytab
--principal=kudu/127.17.53.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.53.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790713339161492-17621-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.17.53.66
--webserver_port=0
--tserver_master_addrs=127.17.53.126:41283
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.53.66
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790713339161492-17621-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:23:07.318245 23484 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:23:07.319679 23484 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:23:07.332902 23489 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:23:07.332950 23490 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:23:07.332978 23492 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:23:07.334061 23484 server_base.cc:1034] running on GCE node
I20260929 20:23:07.334703 23484 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:23:07.335776 23484 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:23:07.337008 23484 hybrid_clock.cc:648] HybridClock initialized: now 1790713387336952 us; error 52 us; skew 500 ppm
I20260929 20:23:07.337332 23484 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:23:07 dist-test-slave-zbnc krb5kdc[23244](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713387, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.53.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:23:07.340592 23484 init.cc:377] Logged in from keytab as kudu/127.17.53.66@KRBTEST.COM (short username kudu)
I20260929 20:23:07.342219 23484 webserver.cc:460] Webserver started at http://127.17.53.66:35083/ using document root <none> and password file <none>
I20260929 20:23:07.343029 23484 fs_manager.cc:359] Metadata directory not provided
I20260929 20:23:07.343175 23484 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:23:07.343498 23484 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:23:07.346438 23484 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790713339161492-17621-0/minicluster-data/ts-1/data/instance:
uuid: "4372b351686947e2bd3e7ed3235c421f"
format_stamp: "Formatted at 2026-09-29 20:23:07 on dist-test-slave-zbnc"
server_key: "c1d8634c5341dfc387a02be2e04f1751"
server_key_iv: "81618d074802000ac5a80a4bc3176789"
server_key_version: "encryptionkey@0"
I20260929 20:23:07.347290 23484 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790713339161492-17621-0/minicluster-data/ts-1/wal/instance:
uuid: "4372b351686947e2bd3e7ed3235c421f"
format_stamp: "Formatted at 2026-09-29 20:23:07 on dist-test-slave-zbnc"
server_key: "c1d8634c5341dfc387a02be2e04f1751"
server_key_iv: "81618d074802000ac5a80a4bc3176789"
server_key_version: "encryptionkey@0"
I20260929 20:23:07.352785 23484 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260929 20:23:07.356426 23499 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:23:07.358000 23484 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260929 20:23:07.358232 23484 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790713339161492-17621-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790713339161492-17621-0/minicluster-data/ts-1/wal
uuid: "4372b351686947e2bd3e7ed3235c421f"
format_stamp: "Formatted at 2026-09-29 20:23:07 on dist-test-slave-zbnc"
server_key: "c1d8634c5341dfc387a02be2e04f1751"
server_key_iv: "81618d074802000ac5a80a4bc3176789"
server_key_version: "encryptionkey@0"
I20260929 20:23:07.358593 23484 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790713339161492-17621-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790713339161492-17621-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790713339161492-17621-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:23:07.376739 23484 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:23:07.381624 23484 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:23:07.381963 23484 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:23:07.383016 23484 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:23:07.384431 23484 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:23:07.384543 23484 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:23:07.384693 23484 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:23:07.384771 23484 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:23:07.420995 23484 rpc_server.cc:307] RPC server started. Bound to: 127.17.53.66:37693
I20260929 20:23:07.421084 23611 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.53.66:37693 every 8 connection(s)
I20260929 20:23:07.422372 23484 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790713339161492-17621-0/minicluster-data/ts-1/data/info.pb
I20260929 20:23:07.427466 17621 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu as pid 23484
I20260929 20:23:07.427651 17621 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790713339161492-17621-0/minicluster-data/ts-1/wal/instance
I20260929 20:23:07.428107 17621 external_mini_cluster.cc:1428] Setting key ebf24966796bf5e9ad8a01c8ca653d7b
Sep 29 20:23:07 dist-test-slave-zbnc krb5kdc[23244](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713387, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.53.66@KRBTEST.COM for kudu/127.17.53.126@KRBTEST.COM
I20260929 20:23:07.445098 23480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:07.425195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.53.66:55769 (local address 127.17.53.126:41283)
0929 20:23:07.425373 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0929 20:23:07.425378 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:23:07.426591 (+ 1213us) server_negotiation.cc:407] Connection header received
0929 20:23:07.428860 (+ 2269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:23:07.428864 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:23:07.428968 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:23:07.429073 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:23:07.431044 (+ 1971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:07.432238 (+ 1194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:07.433873 (+ 1635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:07.434659 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:07.434818 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:07.438780 (+ 3962us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:23:07.438785 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:23:07.438788 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:23:07.438820 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:23:07.441685 (+ 2865us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:23:07.442658 (+ 973us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:23:07.442662 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:23:07.442664 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:23:07.442787 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:23:07.443310 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:23:07.443314 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:23:07.443316 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:23:07.443585 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:23:07.443692 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0929 20:23:07.444663 (+ 971us) server_negotiation.cc:299] Negotiation successful
0929 20:23:07.444834 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260929 20:23:07.446787 23614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:07.425684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:41283 (local address 127.17.53.66:55769)
0929 20:23:07.426415 (+ 731us) negotiation.cc:107] Waiting for socket to connect
0929 20:23:07.426453 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0929 20:23:07.428500 (+ 2047us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:23:07.429374 (+ 874us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:23:07.429387 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:23:07.430132 (+ 745us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:23:07.430818 (+ 686us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:23:07.430835 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:23:07.432409 (+ 1574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:23:07.432413 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:23:07.433717 (+ 1304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:23:07.433728 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:23:07.434912 (+ 1184us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:23:07.434921 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:23:07.435190 (+ 269us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:07.435939 (+ 749us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:23:07.435961 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:23:07.438539 (+ 2578us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:23:07.441835 (+ 3296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:23:07.441848 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:23:07.441869 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:23:07.442521 (+ 652us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:23:07.442928 (+ 407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:23:07.442936 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:23:07.442942 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:23:07.443191 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:23:07.443728 (+ 537us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:23:07.443739 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:23:07.444230 (+ 491us) client_negotiation.cc:769] Sending connection context
0929 20:23:07.444562 (+ 332us) client_negotiation.cc:240] Negotiation successful
0929 20:23:07.445185 (+ 623us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":492,"thread_start_us":306,"threads_started":1}
I20260929 20:23:07.448447 23612 heartbeater.cc:347] Connected to a master server at 127.17.53.126:41283
I20260929 20:23:07.448840 23612 heartbeater.cc:464] Registering TS with master...
I20260929 20:23:07.449690 23612 heartbeater.cc:511] Master 127.17.53.126:41283 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.17.53.67@KRBTEST.COM; defaulting to no policy
I20260929 20:23:07.451335 23289 ts_manager.cc:194] Registered new tserver with Master: 4372b351686947e2bd3e7ed3235c421f (127.17.53.66:37693)
I20260929 20:23:07.452518 23289 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.53.66@KRBTEST.COM'} at 127.17.53.66:55769
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.53.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.53.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.67.keytab.
Entry for principal kudu/127.17.53.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.67.keytab.
I20260929 20:23:07.479043 17621 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
/tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790713339161492-17621-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790713339161492-17621-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790713339161492-17621-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790713339161492-17621-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.53.67:0
--local_ip_for_outbound_sockets=127.17.53.67
--webserver_interface=127.17.53.67
--webserver_port=0
--tserver_master_addrs=127.17.53.126:41283
--keytab_file=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.67.keytab
--principal=kudu/127.17.53.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.53.84:39137
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790713339161492-17621-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790713339161492-17621-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790713339161492-17621-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:23:07.807987 23617 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:23:07.808269 23617 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:23:07.808408 23617 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:23:07.816334 23617 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:23:07.816434 23617 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:23:07.816548 23617 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.53.67
I20260929 20:23:07.826181 23617 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.53.84:39137
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790713339161492-17621-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790713339161492-17621-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.67.keytab
--principal=kudu/127.17.53.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.53.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790713339161492-17621-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.17.53.67
--webserver_port=0
--tserver_master_addrs=127.17.53.126:41283
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.53.67
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790713339161492-17621-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:23:07.827519 23617 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:23:07.828841 23617 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:23:07.842384 23622 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:23:07.842291 23625 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:23:07.842327 23623 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:23:07.843326 23617 server_base.cc:1034] running on GCE node
I20260929 20:23:07.843909 23617 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:23:07.844794 23617 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:23:07.845993 23617 hybrid_clock.cc:648] HybridClock initialized: now 1790713387845930 us; error 52 us; skew 500 ppm
I20260929 20:23:07.846374 23617 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:23:07 dist-test-slave-zbnc krb5kdc[23244](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713387, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.53.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:23:07.849771 23617 init.cc:377] Logged in from keytab as kudu/127.17.53.67@KRBTEST.COM (short username kudu)
I20260929 20:23:07.851362 23617 webserver.cc:460] Webserver started at http://127.17.53.67:41355/ using document root <none> and password file <none>
I20260929 20:23:07.852044 23617 fs_manager.cc:359] Metadata directory not provided
I20260929 20:23:07.852187 23617 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:23:07.852505 23617 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:23:07.855455 23617 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790713339161492-17621-0/minicluster-data/ts-2/data/instance:
uuid: "b33ce02157344a8c8b6351466ef739ae"
format_stamp: "Formatted at 2026-09-29 20:23:07 on dist-test-slave-zbnc"
server_key: "15914955dee3d54bcfc60807006c100c"
server_key_iv: "d855487e77ff6d9356cf4065b913f59c"
server_key_version: "encryptionkey@0"
I20260929 20:23:07.856228 23617 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790713339161492-17621-0/minicluster-data/ts-2/wal/instance:
uuid: "b33ce02157344a8c8b6351466ef739ae"
format_stamp: "Formatted at 2026-09-29 20:23:07 on dist-test-slave-zbnc"
server_key: "15914955dee3d54bcfc60807006c100c"
server_key_iv: "d855487e77ff6d9356cf4065b913f59c"
server_key_version: "encryptionkey@0"
I20260929 20:23:07.861434 23617 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 20:23:07.864826 23632 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:23:07.866196 23617 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 20:23:07.866343 23617 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790713339161492-17621-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790713339161492-17621-0/minicluster-data/ts-2/wal
uuid: "b33ce02157344a8c8b6351466ef739ae"
format_stamp: "Formatted at 2026-09-29 20:23:07 on dist-test-slave-zbnc"
server_key: "15914955dee3d54bcfc60807006c100c"
server_key_iv: "d855487e77ff6d9356cf4065b913f59c"
server_key_version: "encryptionkey@0"
I20260929 20:23:07.866544 23617 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790713339161492-17621-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790713339161492-17621-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790713339161492-17621-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:23:07.886968 23617 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:23:07.891057 23617 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:23:07.891372 23617 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:23:07.892374 23617 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:23:07.893778 23617 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:23:07.893854 23617 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:23:07.894026 23617 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:23:07.894100 23617 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:23:07.927583 23617 rpc_server.cc:307] RPC server started. Bound to: 127.17.53.67:40035
I20260929 20:23:07.927640 23744 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.53.67:40035 every 8 connection(s)
I20260929 20:23:07.929001 23617 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790713339161492-17621-0/minicluster-data/ts-2/data/info.pb
I20260929 20:23:07.930819 17621 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu as pid 23617
I20260929 20:23:07.931048 17621 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790713339161492-17621-0/minicluster-data/ts-2/wal/instance
I20260929 20:23:07.931531 17621 external_mini_cluster.cc:1428] Setting key 3fbb637ff4c9ff61e5ec222d2a463a26
I20260929 20:23:07.938382 23478 heartbeater.cc:503] Master 127.17.53.126:41283 was elected leader, sending a full tablet report...
Sep 29 20:23:07 dist-test-slave-zbnc krb5kdc[23244](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713387, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.53.67@KRBTEST.COM for kudu/127.17.53.126@KRBTEST.COM
I20260929 20:23:07.949769 23480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:07.931613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.53.67:51755 (local address 127.17.53.126:41283)
0929 20:23:07.931820 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0929 20:23:07.931826 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:23:07.933259 (+ 1433us) server_negotiation.cc:407] Connection header received
0929 20:23:07.935396 (+ 2137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:23:07.935400 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:23:07.935490 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:23:07.935615 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:23:07.937003 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:07.937796 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:07.939600 (+ 1804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:07.940190 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:07.940287 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:07.943613 (+ 3326us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:23:07.943618 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:23:07.943621 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:23:07.943651 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:23:07.946707 (+ 3056us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:23:07.947547 (+ 840us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:23:07.947551 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:23:07.947553 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:23:07.947730 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:23:07.948178 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:23:07.948187 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:23:07.948190 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:23:07.948446 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:23:07.948553 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0929 20:23:07.949373 (+ 820us) server_negotiation.cc:299] Negotiation successful
0929 20:23:07.949524 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260929 20:23:07.950812 23747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:07.932231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:41283 (local address 127.17.53.67:51755)
0929 20:23:07.933077 (+ 846us) negotiation.cc:107] Waiting for socket to connect
0929 20:23:07.933135 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0929 20:23:07.935124 (+ 1989us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:23:07.935784 (+ 660us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:23:07.935792 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:23:07.936300 (+ 508us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:23:07.936859 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:23:07.936875 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:23:07.937925 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:23:07.937929 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:23:07.939460 (+ 1531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:23:07.939471 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:23:07.940381 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:23:07.940386 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:23:07.940579 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:07.941263 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:23:07.941288 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:23:07.943429 (+ 2141us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:23:07.946881 (+ 3452us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:23:07.946891 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:23:07.946910 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:23:07.947434 (+ 524us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:23:07.947833 (+ 399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:23:07.947836 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:23:07.947838 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:23:07.948059 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:23:07.948550 (+ 491us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:23:07.948556 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:23:07.948926 (+ 370us) client_negotiation.cc:769] Sending connection context
0929 20:23:07.949197 (+ 271us) client_negotiation.cc:240] Negotiation successful
0929 20:23:07.949585 (+ 388us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":564,"thread_start_us":350,"threads_started":1}
I20260929 20:23:07.952137 23745 heartbeater.cc:347] Connected to a master server at 127.17.53.126:41283
I20260929 20:23:07.952458 23745 heartbeater.cc:464] Registering TS with master...
I20260929 20:23:07.953169 23745 heartbeater.cc:511] Master 127.17.53.126:41283 requested a full tablet report, sending...
I20260929 20:23:07.954517 23289 ts_manager.cc:194] Registered new tserver with Master: b33ce02157344a8c8b6351466ef739ae (127.17.53.67:40035)
I20260929 20:23:07.955386 23289 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.53.67@KRBTEST.COM'} at 127.17.53.67:51755
I20260929 20:23:07.959422 17621 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790713339161492-17621-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790713339161492-17621-0/krb5kdc/test-admin.keytab.
Sep 29 20:23:07 dist-test-slave-zbnc krb5kdc[23244](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713387, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:23:07.986218 17621 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260929 20:23:07.991313 23752 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:23:07.988760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44866 (local address 127.17.53.65:43749)
0929 20:23:07.989403 (+ 643us) server_negotiation.cc:206] Beginning negotiation
0929 20:23:07.989410 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:23:07.989439 (+ 29us) server_negotiation.cc:407] Connection header received
0929 20:23:07.989585 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:23:07.989592 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:23:07.989812 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:23:07.989942 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:23:07.990903 (+ 961us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44866: BlockingRecv error: recv got EOF from 127.0.0.1:44866 (error 108)
Metrics: {"server-negotiator.queue_time_us":504,"thread_start_us":347,"threads_started":1}
W20260929 20:23:07.991375 17621 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790713339161492-17621-0/krb5kdc/krb5cc' of type FILE
Sep 29 20:23:07 dist-test-slave-zbnc krb5kdc[23244](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713387, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:23:07.993018 17621 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 29 20:23:07 dist-test-slave-zbnc krb5kdc[23244](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713387, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:23:07.994375 17621 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 29 20:23:08 dist-test-slave-zbnc krb5kdc[23244](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713387, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.53.65@KRBTEST.COM
I20260929 20:23:08.010107 23752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:07.995985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44880 (local address 127.17.53.65:43749)
0929 20:23:07.996194 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0929 20:23:07.996201 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:23:07.996454 (+ 253us) server_negotiation.cc:407] Connection header received
0929 20:23:07.996705 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:23:07.996709 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:23:07.996766 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:23:07.996873 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:23:07.997865 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:07.998822 (+ 957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:08.000088 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:08.000678 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:08.000815 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:08.004455 (+ 3640us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:23:08.004466 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:23:08.004483 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:23:08.004524 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:23:08.007391 (+ 2867us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:23:08.008048 (+ 657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:23:08.008056 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:23:08.008058 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:23:08.008183 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:23:08.008563 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:23:08.008577 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:23:08.008581 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:23:08.008934 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:23:08.009242 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0929 20:23:08.009579 (+ 337us) server_negotiation.cc:299] Negotiation successful
0929 20:23:08.009834 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
W20260929 20:23:08.016117 23752 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:23:08.014235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44896 (local address 127.17.53.65:43749)
0929 20:23:08.014450 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0929 20:23:08.014456 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:23:08.014727 (+ 271us) server_negotiation.cc:407] Connection header received
0929 20:23:08.014927 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:23:08.014932 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:23:08.015005 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:23:08.015124 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:23:08.015860 (+ 736us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44896: BlockingRecv error: recv got EOF from 127.0.0.1:44896 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
W20260929 20:23:08.016583 17621 init.cc:244] Error while opening credential cache '/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790713339161492-17621-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790713339161492-17621-0/krb5kdc/krb5cc)
Sep 29 20:23:08 dist-test-slave-zbnc krb5kdc[23244](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713388, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:23:08.018093 17621 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 29 20:23:08 dist-test-slave-zbnc krb5kdc[23244](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713388, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:23:08.019408 17621 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 29 20:23:08 dist-test-slave-zbnc krb5kdc[23244](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713388, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.53.65@KRBTEST.COM
I20260929 20:23:08.032868 23752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:08.020813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44904 (local address 127.17.53.65:43749)
0929 20:23:08.020995 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0929 20:23:08.020999 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:23:08.021085 (+ 86us) server_negotiation.cc:407] Connection header received
0929 20:23:08.021208 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:23:08.021212 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:23:08.021265 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:23:08.021341 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:23:08.022198 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:08.022963 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:08.024090 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:08.024513 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:08.024601 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:08.028095 (+ 3494us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:23:08.028099 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:23:08.028102 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:23:08.028145 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:23:08.030670 (+ 2525us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:23:08.031291 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:23:08.031295 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:23:08.031297 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:23:08.031413 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:23:08.031731 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:23:08.031734 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:23:08.031736 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:23:08.031976 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:23:08.032160 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0929 20:23:08.032379 (+ 219us) server_negotiation.cc:299] Negotiation successful
0929 20:23:08.032565 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
W20260929 20:23:08.037256 23752 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:23:08.035631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44914 (local address 127.17.53.65:43749)
0929 20:23:08.035832 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0929 20:23:08.035839 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:23:08.035933 (+ 94us) server_negotiation.cc:407] Connection header received
0929 20:23:08.036142 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:23:08.036147 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:23:08.036225 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:23:08.036327 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:23:08.037023 (+ 696us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44914: BlockingRecv error: recv got EOF from 127.0.0.1:44914 (error 108)
Metrics: {"server-negotiator.queue_time_us":106}
W20260929 20:23:08.037422 17621 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790713339161492-17621-0/krb5kdc/krb5cc' of type FILE
Sep 29 20:23:08 dist-test-slave-zbnc krb5kdc[23244](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713388, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:23:08.039041 17621 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 29 20:23:08 dist-test-slave-zbnc krb5kdc[23244](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713388, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:23:08.040319 17621 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 29 20:23:08 dist-test-slave-zbnc krb5kdc[23244](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713388, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.53.65@KRBTEST.COM
I20260929 20:23:08.055258 23752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:08.041562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44922 (local address 127.17.53.65:43749)
0929 20:23:08.041766 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0929 20:23:08.041774 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 20:23:08.042010 (+ 236us) server_negotiation.cc:407] Connection header received
0929 20:23:08.042209 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:23:08.042212 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:23:08.042265 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:23:08.042366 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:23:08.043362 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:08.044060 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:08.045285 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:08.045699 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:08.045837 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:08.049457 (+ 3620us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:23:08.049462 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:23:08.049465 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:23:08.049494 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:23:08.052805 (+ 3311us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:23:08.053415 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:23:08.053419 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:23:08.053422 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:23:08.053605 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:23:08.053934 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:23:08.053938 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:23:08.053940 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:23:08.054225 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:23:08.054454 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0929 20:23:08.054729 (+ 275us) server_negotiation.cc:299] Negotiation successful
0929 20:23:08.054981 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 20:23:08.057060 17621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu pid 23350
I20260929 20:23:08.068938 23752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:08.057812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44932 (local address 127.17.53.65:43749)
0929 20:23:08.058029 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0929 20:23:08.058034 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:23:08.058275 (+ 241us) server_negotiation.cc:407] Connection header received
0929 20:23:08.058394 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:23:08.058397 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:23:08.058450 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:23:08.058525 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:23:08.059526 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:08.060167 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:08.061466 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:08.061885 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:08.061976 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:08.064334 (+ 2358us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:23:08.064338 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:23:08.064341 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:23:08.064368 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:23:08.066716 (+ 2348us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:23:08.067336 (+ 620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:23:08.067340 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:23:08.067342 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:23:08.067464 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:23:08.067774 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:23:08.067777 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:23:08.067779 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:23:08.067998 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:23:08.068175 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0929 20:23:08.068427 (+ 252us) server_negotiation.cc:299] Negotiation successful
0929 20:23:08.068616 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260929 20:23:08.069489 23473 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:23:08.154063 17621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu with pid 23350
I20260929 20:23:08.171293 17621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu pid 23484
Sep 29 20:23:08 dist-test-slave-zbnc krb5kdc[23244](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713388, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.53.66@KRBTEST.COM
I20260929 20:23:08.187922 23765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:08.172337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35346 (local address 127.17.53.66:37693)
0929 20:23:08.172912 (+ 575us) server_negotiation.cc:206] Beginning negotiation
0929 20:23:08.172919 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:23:08.172945 (+ 26us) server_negotiation.cc:407] Connection header received
0929 20:23:08.173018 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:23:08.173024 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:23:08.173264 (+ 240us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:23:08.173464 (+ 200us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:23:08.174544 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:08.175745 (+ 1201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:08.177089 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:08.177683 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:08.177848 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:08.181872 (+ 4024us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:23:08.181883 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:23:08.181898 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:23:08.181953 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:23:08.185196 (+ 3243us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:23:08.185850 (+ 654us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:23:08.185858 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:23:08.185861 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:23:08.185998 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:23:08.186343 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:23:08.186347 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:23:08.186349 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:23:08.186698 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:23:08.187205 (+ 507us) server_negotiation.cc:1035] Waiting for connection context
0929 20:23:08.187340 (+ 135us) server_negotiation.cc:299] Negotiation successful
0929 20:23:08.187618 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":433,"thread_start_us":297,"threads_started":1}
I20260929 20:23:08.189280 23607 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:23:08.272976 17621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu with pid 23484
I20260929 20:23:08.289330 17621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu pid 23617
Sep 29 20:23:08 dist-test-slave-zbnc krb5kdc[23244](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713388, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.53.67@KRBTEST.COM
I20260929 20:23:08.305853 23767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:08.290476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41056 (local address 127.17.53.67:40035)
0929 20:23:08.291282 (+ 806us) server_negotiation.cc:206] Beginning negotiation
0929 20:23:08.291289 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:23:08.291317 (+ 28us) server_negotiation.cc:407] Connection header received
0929 20:23:08.291375 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:23:08.291381 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:23:08.291583 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:23:08.291777 (+ 194us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:23:08.292875 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:08.293877 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:08.295213 (+ 1336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:08.295754 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:08.295900 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:08.300063 (+ 4163us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:23:08.300075 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:23:08.300092 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:23:08.300133 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:23:08.303055 (+ 2922us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:23:08.303786 (+ 731us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:23:08.303795 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:23:08.303797 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:23:08.303921 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:23:08.304305 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:23:08.304312 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:23:08.304315 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:23:08.304660 (+ 345us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:23:08.304977 (+ 317us) server_negotiation.cc:1035] Waiting for connection context
0929 20:23:08.305285 (+ 308us) server_negotiation.cc:299] Negotiation successful
0929 20:23:08.305548 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":634,"thread_start_us":503,"threads_started":1}
I20260929 20:23:08.307312 23740 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:23:08.390965 17621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu with pid 23617
I20260929 20:23:08.406947 17621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu pid 23257
I20260929 20:23:08.408159 23319 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:23:08.513593 17621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu with pid 23257
2026-09-29T20:23:08Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (3999 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713339161492-17621-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 20:23:08 dist-test-slave-zbnc krb5kdc[23772](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 20:23:08 dist-test-slave-zbnc krb5kdc[23772](info): set up 2 sockets
Sep 29 20:23:08 dist-test-slave-zbnc krb5kdc[23772](info): commencing operation
krb5kdc: starting...
W20260929 20:23:09.617027 17621 mini_kdc.cc:121] Time spent starting KDC: real 1.061s user 0.003s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713339161492-17621-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713339161492-17621-0/krb5kdc/test-user.keytab.
Sep 29 20:23:09 dist-test-slave-zbnc krb5kdc[23772](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713389, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T20:23:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T20:23:09Z Disabled control of system clock
WARNING: no policy specified for oryx/127.17.53.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.17.53.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.17.53.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713339161492-17621-0/krb5kdc/oryx_127.17.53.126.keytab.
Entry for principal oryx/127.17.53.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713339161492-17621-0/krb5kdc/oryx_127.17.53.126.keytab.
I20260929 20:23:09.834702 17621 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
/tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713339161492-17621-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713339161492-17621-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713339161492-17621-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713339161492-17621-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.53.126:34513
--webserver_interface=127.17.53.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713339161492-17621-0/krb5kdc/oryx_127.17.53.126.keytab
--principal=oryx/127.17.53.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.53.84:41905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.53.126:34513
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713339161492-17621-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713339161492-17621-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713339161492-17621-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:23:10.160832 23785 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:23:10.161116 23785 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:23:10.161239 23785 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:23:10.168372 23785 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:23:10.168478 23785 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:23:10.168596 23785 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:23:10.168692 23785 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:23:10.168779 23785 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:23:10.178522 23785 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.53.84:41905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713339161492-17621-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713339161492-17621-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.53.126:34513
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713339161492-17621-0/krb5kdc/oryx_127.17.53.126.keytab
--principal=oryx/127.17.53.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.53.126:34513
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713339161492-17621-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.53.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713339161492-17621-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:23:10.179968 23785 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:23:10.181283 23785 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:23:10.193791 23790 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:23:10.193749 23791 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:23:10.193872 23793 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:23:10.194604 23785 server_base.cc:1034] running on GCE node
I20260929 20:23:10.195447 23785 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:23:10.196864 23785 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:23:10.198099 23785 hybrid_clock.cc:648] HybridClock initialized: now 1790713390198058 us; error 49 us; skew 500 ppm
I20260929 20:23:10.198460 23785 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:23:10 dist-test-slave-zbnc krb5kdc[23772](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713390, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.53.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:23:10.201521 23785 init.cc:377] Logged in from keytab as oryx/127.17.53.126@KRBTEST.COM (short username oryx)
I20260929 20:23:10.203229 23785 webserver.cc:460] Webserver started at http://127.17.53.126:45407/ using document root <none> and password file <none>
I20260929 20:23:10.203927 23785 fs_manager.cc:359] Metadata directory not provided
I20260929 20:23:10.204073 23785 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:23:10.204388 23785 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:23:10.207490 23785 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713339161492-17621-0/minicluster-data/master-0/data/instance:
uuid: "94c779fe1aa149a9add4b5b63f72b2df"
format_stamp: "Formatted at 2026-09-29 20:23:10 on dist-test-slave-zbnc"
server_key: "f351efe0aaaa52b9d4fca0f581e7655b"
server_key_iv: "89e8da33379edd7d69e9ae26aff06427"
server_key_version: "encryptionkey@0"
I20260929 20:23:10.208268 23785 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713339161492-17621-0/minicluster-data/master-0/wal/instance:
uuid: "94c779fe1aa149a9add4b5b63f72b2df"
format_stamp: "Formatted at 2026-09-29 20:23:10 on dist-test-slave-zbnc"
server_key: "f351efe0aaaa52b9d4fca0f581e7655b"
server_key_iv: "89e8da33379edd7d69e9ae26aff06427"
server_key_version: "encryptionkey@0"
I20260929 20:23:10.213601 23785 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.002s
I20260929 20:23:10.217013 23800 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:23:10.218356 23785 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 20:23:10.218552 23785 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713339161492-17621-0/minicluster-data/master-0/data,/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713339161492-17621-0/minicluster-data/master-0/wal
uuid: "94c779fe1aa149a9add4b5b63f72b2df"
format_stamp: "Formatted at 2026-09-29 20:23:10 on dist-test-slave-zbnc"
server_key: "f351efe0aaaa52b9d4fca0f581e7655b"
server_key_iv: "89e8da33379edd7d69e9ae26aff06427"
server_key_version: "encryptionkey@0"
I20260929 20:23:10.218765 23785 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713339161492-17621-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713339161492-17621-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713339161492-17621-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:23:10.245484 23785 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:23:10.250085 23785 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:23:10.250505 23785 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:23:10.269361 23785 rpc_server.cc:307] RPC server started. Bound to: 127.17.53.126:34513
I20260929 20:23:10.269382 23851 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.53.126:34513 every 8 connection(s)
I20260929 20:23:10.270771 23785 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713339161492-17621-0/minicluster-data/master-0/data/info.pb
I20260929 20:23:10.274446 17621 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu as pid 23785
I20260929 20:23:10.274663 17621 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713339161492-17621-0/minicluster-data/master-0/wal/instance
I20260929 20:23:10.274802 23852 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:23:10.275405 17621 external_mini_cluster.cc:1428] Setting key d97bc5ca80807893fed68adfabcd4f71
I20260929 20:23:10.289089 23852 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 94c779fe1aa149a9add4b5b63f72b2df: Bootstrap starting.
Sep 29 20:23:10 dist-test-slave-zbnc krb5kdc[23772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713389, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.17.53.126@KRBTEST.COM
I20260929 20:23:10.293572 23852 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 94c779fe1aa149a9add4b5b63f72b2df: Neither blocks nor log segments found. Creating new log.
I20260929 20:23:10.295074 23852 log.cc:824] T 00000000000000000000000000000000 P 94c779fe1aa149a9add4b5b63f72b2df: Log is configured to *not* fsync() on all Append() calls
I20260929 20:23:10.298777 23852 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 94c779fe1aa149a9add4b5b63f72b2df: No bootstrap required, opened a new log
I20260929 20:23:10.300238 23855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:10.277367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49342 (local address 127.17.53.126:34513)
0929 20:23:10.278168 (+ 801us) server_negotiation.cc:206] Beginning negotiation
0929 20:23:10.278187 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0929 20:23:10.278240 (+ 53us) server_negotiation.cc:407] Connection header received
0929 20:23:10.280074 (+ 1834us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:23:10.280107 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:23:10.280683 (+ 576us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:23:10.281145 (+ 462us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:23:10.282494 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:10.284419 (+ 1925us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:10.285639 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:10.286748 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:10.286961 (+ 213us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:10.290948 (+ 3987us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:23:10.290960 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:23:10.290982 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:23:10.291045 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:23:10.295221 (+ 4176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:23:10.296149 (+ 928us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:23:10.296157 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:23:10.296164 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:23:10.296363 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:23:10.296883 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:23:10.296891 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:23:10.296897 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:23:10.297600 (+ 703us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:23:10.297935 (+ 335us) server_negotiation.cc:1035] Waiting for connection context
0929 20:23:10.298290 (+ 355us) server_negotiation.cc:299] Negotiation successful
0929 20:23:10.298727 (+ 437us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":555,"thread_start_us":332,"threads_started":1}
I20260929 20:23:10.308202 23852 raft_consensus.cc:348] T 00000000000000000000000000000000 P 94c779fe1aa149a9add4b5b63f72b2df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94c779fe1aa149a9add4b5b63f72b2df" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 34513 } }
I20260929 20:23:10.308619 23852 raft_consensus.cc:374] T 00000000000000000000000000000000 P 94c779fe1aa149a9add4b5b63f72b2df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:23:10.308706 23852 raft_consensus.cc:729] T 00000000000000000000000000000000 P 94c779fe1aa149a9add4b5b63f72b2df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94c779fe1aa149a9add4b5b63f72b2df, State: Initialized, Role: FOLLOWER
I20260929 20:23:10.309329 23852 consensus_queue.cc:260] T 00000000000000000000000000000000 P 94c779fe1aa149a9add4b5b63f72b2df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94c779fe1aa149a9add4b5b63f72b2df" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 34513 } }
I20260929 20:23:10.309603 23852 raft_consensus.cc:388] T 00000000000000000000000000000000 P 94c779fe1aa149a9add4b5b63f72b2df [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 20:23:10.309782 23852 raft_consensus.cc:482] T 00000000000000000000000000000000 P 94c779fe1aa149a9add4b5b63f72b2df [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 20:23:10.309969 23852 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 94c779fe1aa149a9add4b5b63f72b2df [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:23:10.312340 23852 raft_consensus.cc:504] T 00000000000000000000000000000000 P 94c779fe1aa149a9add4b5b63f72b2df [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94c779fe1aa149a9add4b5b63f72b2df" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 34513 } }
I20260929 20:23:10.312887 23852 leader_election.cc:302] T 00000000000000000000000000000000 P 94c779fe1aa149a9add4b5b63f72b2df [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 94c779fe1aa149a9add4b5b63f72b2df; no voters:
I20260929 20:23:10.313557 23852 leader_election.cc:288] T 00000000000000000000000000000000 P 94c779fe1aa149a9add4b5b63f72b2df [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 20:23:10.313694 23857 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 94c779fe1aa149a9add4b5b63f72b2df [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:23:10.314468 23857 raft_consensus.cc:686] T 00000000000000000000000000000000 P 94c779fe1aa149a9add4b5b63f72b2df [term 1 LEADER]: Becoming Leader. State: Replica: 94c779fe1aa149a9add4b5b63f72b2df, State: Running, Role: LEADER
I20260929 20:23:10.315030 23857 consensus_queue.cc:237] T 00000000000000000000000000000000 P 94c779fe1aa149a9add4b5b63f72b2df [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94c779fe1aa149a9add4b5b63f72b2df" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 34513 } }
I20260929 20:23:10.315857 23852 sys_catalog.cc:564] T 00000000000000000000000000000000 P 94c779fe1aa149a9add4b5b63f72b2df [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:23:10.318269 23859 sys_catalog.cc:455] T 00000000000000000000000000000000 P 94c779fe1aa149a9add4b5b63f72b2df [sys.catalog]: SysCatalogTable state changed. Reason: New leader 94c779fe1aa149a9add4b5b63f72b2df. Latest consensus state: current_term: 1 leader_uuid: "94c779fe1aa149a9add4b5b63f72b2df" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94c779fe1aa149a9add4b5b63f72b2df" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 34513 } } }
I20260929 20:23:10.318714 23859 sys_catalog.cc:458] T 00000000000000000000000000000000 P 94c779fe1aa149a9add4b5b63f72b2df [sys.catalog]: This master's current role is: LEADER
I20260929 20:23:10.319556 23858 sys_catalog.cc:455] T 00000000000000000000000000000000 P 94c779fe1aa149a9add4b5b63f72b2df [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "94c779fe1aa149a9add4b5b63f72b2df" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94c779fe1aa149a9add4b5b63f72b2df" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 34513 } } }
I20260929 20:23:10.319912 23858 sys_catalog.cc:458] T 00000000000000000000000000000000 P 94c779fe1aa149a9add4b5b63f72b2df [sys.catalog]: This master's current role is: LEADER
I20260929 20:23:10.320499 23864 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:23:10.326085 23864 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:23:10.336783 23864 catalog_manager.cc:1347] Generated new cluster ID: 8dd81d8e76ec4d3d8e283d6964a8f856
I20260929 20:23:10.336931 23864 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:23:10.351243 23864 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 20:23:10.352612 23864 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:23:10.360616 23864 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 94c779fe1aa149a9add4b5b63f72b2df: Generated new TSK 0
I20260929 20:23:10.361469 23864 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.17.53.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.17.53.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.17.53.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713339161492-17621-0/krb5kdc/oryx_127.17.53.65.keytab.
Entry for principal oryx/127.17.53.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713339161492-17621-0/krb5kdc/oryx_127.17.53.65.keytab.
I20260929 20:23:10.423424 17621 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
/tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713339161492-17621-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713339161492-17621-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713339161492-17621-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713339161492-17621-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.53.65:0
--local_ip_for_outbound_sockets=127.17.53.65
--webserver_interface=127.17.53.65
--webserver_port=0
--tserver_master_addrs=127.17.53.126:34513
--keytab_file=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713339161492-17621-0/krb5kdc/oryx_127.17.53.65.keytab
--principal=oryx/127.17.53.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.53.84:41905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713339161492-17621-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713339161492-17621-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713339161492-17621-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:23:10.766213 23878 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:23:10.766573 23878 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:23:10.766755 23878 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:23:10.774415 23878 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:23:10.774582 23878 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:23:10.774770 23878 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.53.65
I20260929 20:23:10.784792 23878 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.53.84:41905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713339161492-17621-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713339161492-17621-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713339161492-17621-0/krb5kdc/oryx_127.17.53.65.keytab
--principal=oryx/127.17.53.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.53.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713339161492-17621-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.17.53.65
--webserver_port=0
--tserver_master_addrs=127.17.53.126:34513
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.53.65
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713339161492-17621-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:23:10.786309 23878 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:23:10.787708 23878 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:23:10.801213 23884 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:23:10.801306 23883 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:23:10.801337 23886 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:23:10.802336 23878 server_base.cc:1034] running on GCE node
I20260929 20:23:10.803110 23878 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:23:10.804082 23878 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:23:10.805286 23878 hybrid_clock.cc:648] HybridClock initialized: now 1790713390805224 us; error 70 us; skew 500 ppm
I20260929 20:23:10.805721 23878 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:23:10 dist-test-slave-zbnc krb5kdc[23772](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713390, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.53.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:23:10.809063 23878 init.cc:377] Logged in from keytab as oryx/127.17.53.65@KRBTEST.COM (short username oryx)
I20260929 20:23:10.810627 23878 webserver.cc:460] Webserver started at http://127.17.53.65:36325/ using document root <none> and password file <none>
I20260929 20:23:10.811414 23878 fs_manager.cc:359] Metadata directory not provided
I20260929 20:23:10.811524 23878 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:23:10.811823 23878 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:23:10.814797 23878 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713339161492-17621-0/minicluster-data/ts-0/data/instance:
uuid: "9bfa2167798b44b1aeb4934b00d01783"
format_stamp: "Formatted at 2026-09-29 20:23:10 on dist-test-slave-zbnc"
server_key: "a7696d9cca47ee27e22753f1870c3495"
server_key_iv: "6d8668926670c223f2f77619efadde53"
server_key_version: "encryptionkey@0"
I20260929 20:23:10.815737 23878 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713339161492-17621-0/minicluster-data/ts-0/wal/instance:
uuid: "9bfa2167798b44b1aeb4934b00d01783"
format_stamp: "Formatted at 2026-09-29 20:23:10 on dist-test-slave-zbnc"
server_key: "a7696d9cca47ee27e22753f1870c3495"
server_key_iv: "6d8668926670c223f2f77619efadde53"
server_key_version: "encryptionkey@0"
I20260929 20:23:10.821324 23878 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 20:23:10.825107 23893 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:23:10.826586 23878 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 20:23:10.826750 23878 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713339161492-17621-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713339161492-17621-0/minicluster-data/ts-0/wal
uuid: "9bfa2167798b44b1aeb4934b00d01783"
format_stamp: "Formatted at 2026-09-29 20:23:10 on dist-test-slave-zbnc"
server_key: "a7696d9cca47ee27e22753f1870c3495"
server_key_iv: "6d8668926670c223f2f77619efadde53"
server_key_version: "encryptionkey@0"
I20260929 20:23:10.826999 23878 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713339161492-17621-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713339161492-17621-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713339161492-17621-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:23:10.846756 23878 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:23:10.851166 23878 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:23:10.851454 23878 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:23:10.852432 23878 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:23:10.854110 23878 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:23:10.854188 23878 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:23:10.854461 23878 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:23:10.854545 23878 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:23:10.890231 23878 rpc_server.cc:307] RPC server started. Bound to: 127.17.53.65:46815
I20260929 20:23:10.890336 24005 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.53.65:46815 every 8 connection(s)
I20260929 20:23:10.891656 23878 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713339161492-17621-0/minicluster-data/ts-0/data/info.pb
I20260929 20:23:10.897387 17621 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu as pid 23878
I20260929 20:23:10.897599 17621 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713339161492-17621-0/minicluster-data/ts-0/wal/instance
I20260929 20:23:10.898456 17621 external_mini_cluster.cc:1428] Setting key 8d4347b6e06dc40dc80d79dbad261ebf
Sep 29 20:23:10 dist-test-slave-zbnc krb5kdc[23772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713390, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.53.65@KRBTEST.COM for oryx/127.17.53.126@KRBTEST.COM
I20260929 20:23:10.916575 24008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:10.894306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.53.65:57601 (local address 127.17.53.126:34513)
0929 20:23:10.894781 (+ 475us) server_negotiation.cc:206] Beginning negotiation
0929 20:23:10.894785 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:23:10.895879 (+ 1094us) server_negotiation.cc:407] Connection header received
0929 20:23:10.898184 (+ 2305us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:23:10.898188 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:23:10.898259 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:23:10.898360 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:23:10.900335 (+ 1975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:10.901196 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:10.903801 (+ 2605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:10.904192 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:10.904294 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:10.908937 (+ 4643us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:23:10.908945 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:23:10.908953 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:23:10.908997 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:23:10.912906 (+ 3909us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:23:10.913695 (+ 789us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:23:10.913705 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:23:10.913711 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:23:10.913874 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:23:10.914315 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:23:10.914319 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:23:10.914321 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:23:10.914831 (+ 510us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:23:10.915038 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0929 20:23:10.915735 (+ 697us) server_negotiation.cc:299] Negotiation successful
0929 20:23:10.916009 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":371,"thread_start_us":252,"threads_started":1}
I20260929 20:23:10.917563 24009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:10.895022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:34513 (local address 127.17.53.65:57601)
0929 20:23:10.895713 (+ 691us) negotiation.cc:107] Waiting for socket to connect
0929 20:23:10.895757 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 20:23:10.897839 (+ 2082us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:23:10.898663 (+ 824us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:23:10.898675 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:23:10.899426 (+ 751us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:23:10.900094 (+ 668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:23:10.900112 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:23:10.901349 (+ 1237us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:23:10.901357 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:23:10.903623 (+ 2266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:23:10.903640 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:23:10.904429 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:23:10.904438 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:23:10.904774 (+ 336us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:10.905533 (+ 759us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:23:10.905574 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:23:10.908626 (+ 3052us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:23:10.913067 (+ 4441us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:23:10.913074 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:23:10.913089 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:23:10.913525 (+ 436us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:23:10.914013 (+ 488us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:23:10.914016 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:23:10.914019 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:23:10.914198 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:23:10.914990 (+ 792us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:23:10.914997 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:23:10.915325 (+ 328us) client_negotiation.cc:769] Sending connection context
0929 20:23:10.915558 (+ 233us) client_negotiation.cc:240] Negotiation successful
0929 20:23:10.916123 (+ 565us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":257,"threads_started":1}
I20260929 20:23:10.919376 24006 heartbeater.cc:347] Connected to a master server at 127.17.53.126:34513
I20260929 20:23:10.919725 24006 heartbeater.cc:464] Registering TS with master...
I20260929 20:23:10.920573 24006 heartbeater.cc:511] Master 127.17.53.126:34513 requested a full tablet report, sending...
WARNING: no policy specified for oryx/127.17.53.66@KRBTEST.COM; defaulting to no policy
I20260929 20:23:10.923041 23817 ts_manager.cc:194] Registered new tserver with Master: 9bfa2167798b44b1aeb4934b00d01783 (127.17.53.65:46815)
I20260929 20:23:10.924625 23817 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.17.53.65@KRBTEST.COM'} at 127.17.53.65:57601
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.17.53.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.17.53.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713339161492-17621-0/krb5kdc/oryx_127.17.53.66.keytab.
Entry for principal oryx/127.17.53.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713339161492-17621-0/krb5kdc/oryx_127.17.53.66.keytab.
I20260929 20:23:10.951099 17621 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
/tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713339161492-17621-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713339161492-17621-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713339161492-17621-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713339161492-17621-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.53.66:0
--local_ip_for_outbound_sockets=127.17.53.66
--webserver_interface=127.17.53.66
--webserver_port=0
--tserver_master_addrs=127.17.53.126:34513
--keytab_file=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713339161492-17621-0/krb5kdc/oryx_127.17.53.66.keytab
--principal=oryx/127.17.53.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.53.84:41905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713339161492-17621-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713339161492-17621-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713339161492-17621-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:23:11.303792 24012 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:23:11.304077 24012 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:23:11.304253 24012 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:23:11.311581 24012 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:23:11.311685 24012 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:23:11.311893 24012 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.53.66
I20260929 20:23:11.322082 24012 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.53.84:41905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713339161492-17621-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713339161492-17621-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713339161492-17621-0/krb5kdc/oryx_127.17.53.66.keytab
--principal=oryx/127.17.53.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.53.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713339161492-17621-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.17.53.66
--webserver_port=0
--tserver_master_addrs=127.17.53.126:34513
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.53.66
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713339161492-17621-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:23:11.323577 24012 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:23:11.325009 24012 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:23:11.338138 24020 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:23:11.338227 24017 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:23:11.338301 24018 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:23:11.338982 24012 server_base.cc:1034] running on GCE node
I20260929 20:23:11.339540 24012 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:23:11.340390 24012 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:23:11.341581 24012 hybrid_clock.cc:648] HybridClock initialized: now 1790713391341509 us; error 61 us; skew 500 ppm
I20260929 20:23:11.341888 24012 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:23:11 dist-test-slave-zbnc krb5kdc[23772](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713391, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.53.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:23:11.349124 24012 init.cc:377] Logged in from keytab as oryx/127.17.53.66@KRBTEST.COM (short username oryx)
I20260929 20:23:11.350561 24012 webserver.cc:460] Webserver started at http://127.17.53.66:38109/ using document root <none> and password file <none>
I20260929 20:23:11.351306 24012 fs_manager.cc:359] Metadata directory not provided
I20260929 20:23:11.351435 24012 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:23:11.351742 24012 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:23:11.354650 24012 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713339161492-17621-0/minicluster-data/ts-1/data/instance:
uuid: "458a0c68846640e7b375f7500abe9076"
format_stamp: "Formatted at 2026-09-29 20:23:11 on dist-test-slave-zbnc"
server_key: "c15f13e749933846e0f03b63f8379ff2"
server_key_iv: "ae93643beefc362efa1be18970a213a8"
server_key_version: "encryptionkey@0"
I20260929 20:23:11.355482 24012 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713339161492-17621-0/minicluster-data/ts-1/wal/instance:
uuid: "458a0c68846640e7b375f7500abe9076"
format_stamp: "Formatted at 2026-09-29 20:23:11 on dist-test-slave-zbnc"
server_key: "c15f13e749933846e0f03b63f8379ff2"
server_key_iv: "ae93643beefc362efa1be18970a213a8"
server_key_version: "encryptionkey@0"
I20260929 20:23:11.360471 24012 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260929 20:23:11.363886 24027 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:23:11.365269 24012 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 20:23:11.365432 24012 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713339161492-17621-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713339161492-17621-0/minicluster-data/ts-1/wal
uuid: "458a0c68846640e7b375f7500abe9076"
format_stamp: "Formatted at 2026-09-29 20:23:11 on dist-test-slave-zbnc"
server_key: "c15f13e749933846e0f03b63f8379ff2"
server_key_iv: "ae93643beefc362efa1be18970a213a8"
server_key_version: "encryptionkey@0"
I20260929 20:23:11.365628 24012 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713339161492-17621-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713339161492-17621-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713339161492-17621-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:23:11.383644 24012 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:23:11.387888 24012 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:23:11.388164 24012 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:23:11.389106 24012 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:23:11.390475 24012 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:23:11.390547 24012 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:23:11.390674 24012 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:23:11.390731 24012 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:23:11.425015 24012 rpc_server.cc:307] RPC server started. Bound to: 127.17.53.66:35805
I20260929 20:23:11.425062 24139 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.53.66:35805 every 8 connection(s)
I20260929 20:23:11.426304 24012 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713339161492-17621-0/minicluster-data/ts-1/data/info.pb
I20260929 20:23:11.431321 17621 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu as pid 24012
I20260929 20:23:11.431546 17621 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713339161492-17621-0/minicluster-data/ts-1/wal/instance
I20260929 20:23:11.432118 17621 external_mini_cluster.cc:1428] Setting key eb7539cd63b9126ccada1149d21db5d8
Sep 29 20:23:11 dist-test-slave-zbnc krb5kdc[23772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713391, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.53.66@KRBTEST.COM for oryx/127.17.53.126@KRBTEST.COM
I20260929 20:23:11.450155 24142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:11.428897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.53.66:53871 (local address 127.17.53.126:34513)
0929 20:23:11.429361 (+ 464us) server_negotiation.cc:206] Beginning negotiation
0929 20:23:11.429365 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:23:11.430487 (+ 1122us) server_negotiation.cc:407] Connection header received
0929 20:23:11.433035 (+ 2548us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:23:11.433041 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:23:11.433135 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:23:11.433274 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:23:11.435094 (+ 1820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:11.435817 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:11.438315 (+ 2498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:11.438731 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:11.438822 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:11.442812 (+ 3990us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:23:11.442821 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:23:11.442829 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:23:11.442898 (+ 69us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:23:11.446497 (+ 3599us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:23:11.447410 (+ 913us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:23:11.447418 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:23:11.447425 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:23:11.447584 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:23:11.448026 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:23:11.448034 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:23:11.448040 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:23:11.448434 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:23:11.448587 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0929 20:23:11.449404 (+ 817us) server_negotiation.cc:299] Negotiation successful
0929 20:23:11.449652 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":360,"thread_start_us":251,"threads_started":1}
I20260929 20:23:11.450779 24143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:11.429579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:34513 (local address 127.17.53.66:53871)
0929 20:23:11.430321 (+ 742us) negotiation.cc:107] Waiting for socket to connect
0929 20:23:11.430379 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0929 20:23:11.432749 (+ 2370us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:23:11.433521 (+ 772us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:23:11.433530 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:23:11.434074 (+ 544us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:23:11.434912 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:23:11.434941 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:23:11.435979 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:23:11.435987 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:23:11.438164 (+ 2177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:23:11.438183 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:23:11.438983 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:23:11.438992 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:23:11.439342 (+ 350us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:11.440092 (+ 750us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:23:11.440133 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:23:11.442562 (+ 2429us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:23:11.446667 (+ 4105us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:23:11.446675 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:23:11.446690 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:23:11.447266 (+ 576us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:23:11.447697 (+ 431us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:23:11.447701 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:23:11.447703 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:23:11.447904 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:23:11.448588 (+ 684us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:23:11.448596 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:23:11.449003 (+ 407us) client_negotiation.cc:769] Sending connection context
0929 20:23:11.449263 (+ 260us) client_negotiation.cc:240] Negotiation successful
0929 20:23:11.449650 (+ 387us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":452,"thread_start_us":259,"threads_started":1}
I20260929 20:23:11.452396 24140 heartbeater.cc:347] Connected to a master server at 127.17.53.126:34513
I20260929 20:23:11.452713 24140 heartbeater.cc:464] Registering TS with master...
I20260929 20:23:11.453449 24140 heartbeater.cc:511] Master 127.17.53.126:34513 requested a full tablet report, sending...
WARNING: no policy specified for oryx/127.17.53.67@KRBTEST.COM; defaulting to no policy
I20260929 20:23:11.455008 23817 ts_manager.cc:194] Registered new tserver with Master: 458a0c68846640e7b375f7500abe9076 (127.17.53.66:35805)
I20260929 20:23:11.455864 23817 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.17.53.66@KRBTEST.COM'} at 127.17.53.66:53871
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.17.53.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.17.53.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713339161492-17621-0/krb5kdc/oryx_127.17.53.67.keytab.
Entry for principal oryx/127.17.53.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713339161492-17621-0/krb5kdc/oryx_127.17.53.67.keytab.
I20260929 20:23:11.481467 17621 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
/tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713339161492-17621-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713339161492-17621-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713339161492-17621-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713339161492-17621-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.53.67:0
--local_ip_for_outbound_sockets=127.17.53.67
--webserver_interface=127.17.53.67
--webserver_port=0
--tserver_master_addrs=127.17.53.126:34513
--keytab_file=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713339161492-17621-0/krb5kdc/oryx_127.17.53.67.keytab
--principal=oryx/127.17.53.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.53.84:41905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713339161492-17621-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713339161492-17621-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713339161492-17621-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:23:11.803285 24146 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:23:11.803604 24146 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:23:11.803788 24146 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:23:11.811031 24146 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:23:11.811190 24146 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:23:11.811384 24146 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.53.67
I20260929 20:23:11.821462 24146 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.53.84:41905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713339161492-17621-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713339161492-17621-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713339161492-17621-0/krb5kdc/oryx_127.17.53.67.keytab
--principal=oryx/127.17.53.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.53.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713339161492-17621-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.17.53.67
--webserver_port=0
--tserver_master_addrs=127.17.53.126:34513
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.53.67
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713339161492-17621-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:23:11.822791 24146 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:23:11.824234 24146 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:23:11.839290 24151 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:23:11.839252 24154 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:23:11.839315 24152 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:23:11.840186 24146 server_base.cc:1034] running on GCE node
I20260929 20:23:11.840821 24146 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:23:11.841781 24146 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:23:11.842978 24146 hybrid_clock.cc:648] HybridClock initialized: now 1790713391842921 us; error 48 us; skew 500 ppm
I20260929 20:23:11.843317 24146 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:23:11 dist-test-slave-zbnc krb5kdc[23772](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713391, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.53.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:23:11.846325 24146 init.cc:377] Logged in from keytab as oryx/127.17.53.67@KRBTEST.COM (short username oryx)
I20260929 20:23:11.847880 24146 webserver.cc:460] Webserver started at http://127.17.53.67:35187/ using document root <none> and password file <none>
I20260929 20:23:11.848578 24146 fs_manager.cc:359] Metadata directory not provided
I20260929 20:23:11.848726 24146 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:23:11.849031 24146 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:23:11.852003 24146 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713339161492-17621-0/minicluster-data/ts-2/data/instance:
uuid: "cd595f79664a402192976b9a664b71e7"
format_stamp: "Formatted at 2026-09-29 20:23:11 on dist-test-slave-zbnc"
server_key: "6f7e145f6b441674d81b4fa8bf810520"
server_key_iv: "48ba61e516b2be24afd24995eaa8110f"
server_key_version: "encryptionkey@0"
I20260929 20:23:11.852798 24146 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713339161492-17621-0/minicluster-data/ts-2/wal/instance:
uuid: "cd595f79664a402192976b9a664b71e7"
format_stamp: "Formatted at 2026-09-29 20:23:11 on dist-test-slave-zbnc"
server_key: "6f7e145f6b441674d81b4fa8bf810520"
server_key_iv: "48ba61e516b2be24afd24995eaa8110f"
server_key_version: "encryptionkey@0"
I20260929 20:23:11.857906 24146 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 20:23:11.861413 24161 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:23:11.862900 24146 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 20:23:11.863080 24146 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713339161492-17621-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713339161492-17621-0/minicluster-data/ts-2/wal
uuid: "cd595f79664a402192976b9a664b71e7"
format_stamp: "Formatted at 2026-09-29 20:23:11 on dist-test-slave-zbnc"
server_key: "6f7e145f6b441674d81b4fa8bf810520"
server_key_iv: "48ba61e516b2be24afd24995eaa8110f"
server_key_version: "encryptionkey@0"
I20260929 20:23:11.863296 24146 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713339161492-17621-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713339161492-17621-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713339161492-17621-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:23:11.881552 24146 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:23:11.885583 24146 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:23:11.885892 24146 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:23:11.886910 24146 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:23:11.888500 24146 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:23:11.888602 24146 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:23:11.888760 24146 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:23:11.888836 24146 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:23:11.923080 24146 rpc_server.cc:307] RPC server started. Bound to: 127.17.53.67:46647
I20260929 20:23:11.923144 24273 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.53.67:46647 every 8 connection(s)
I20260929 20:23:11.924392 24146 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713339161492-17621-0/minicluster-data/ts-2/data/info.pb
I20260929 20:23:11.927877 24006 heartbeater.cc:503] Master 127.17.53.126:34513 was elected leader, sending a full tablet report...
I20260929 20:23:11.930188 17621 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu as pid 24146
I20260929 20:23:11.930370 17621 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713339161492-17621-0/minicluster-data/ts-2/wal/instance
I20260929 20:23:11.931015 17621 external_mini_cluster.cc:1428] Setting key 45543e75416e3c5ef231658295ab2f0a
Sep 29 20:23:11 dist-test-slave-zbnc krb5kdc[23772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713391, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.53.67@KRBTEST.COM for oryx/127.17.53.126@KRBTEST.COM
I20260929 20:23:11.944397 24142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:11.926917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.53.67:56901 (local address 127.17.53.126:34513)
0929 20:23:11.927102 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0929 20:23:11.927106 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:23:11.928430 (+ 1324us) server_negotiation.cc:407] Connection header received
0929 20:23:11.930824 (+ 2394us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:23:11.930831 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:23:11.930943 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:23:11.931024 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:23:11.932785 (+ 1761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:11.933882 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:11.935578 (+ 1696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:11.935952 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:11.936042 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:11.939436 (+ 3394us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:23:11.939441 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:23:11.939445 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:23:11.939475 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:23:11.941823 (+ 2348us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:23:11.942488 (+ 665us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:23:11.942491 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:23:11.942493 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:23:11.942596 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:23:11.943020 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:23:11.943023 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:23:11.943025 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:23:11.943290 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:23:11.943402 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0929 20:23:11.944012 (+ 610us) server_negotiation.cc:299] Negotiation successful
0929 20:23:11.944165 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 20:23:11.945662 24276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:11.927449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:34513 (local address 127.17.53.67:56901)
0929 20:23:11.928271 (+ 822us) negotiation.cc:107] Waiting for socket to connect
0929 20:23:11.928315 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 20:23:11.930529 (+ 2214us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:23:11.931326 (+ 797us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:23:11.931339 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:23:11.931916 (+ 577us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:23:11.932603 (+ 687us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:23:11.932631 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:23:11.934085 (+ 1454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:23:11.934092 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:23:11.935449 (+ 1357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:23:11.935461 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:23:11.936152 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:23:11.936157 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:23:11.936350 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:11.937037 (+ 687us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:23:11.937071 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:23:11.939247 (+ 2176us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:23:11.941947 (+ 2700us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:23:11.941954 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:23:11.941967 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:23:11.942382 (+ 415us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:23:11.942688 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:23:11.942692 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:23:11.942694 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:23:11.942912 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:23:11.943393 (+ 481us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:23:11.943399 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:23:11.943706 (+ 307us) client_negotiation.cc:769] Sending connection context
0929 20:23:11.943923 (+ 217us) client_negotiation.cc:240] Negotiation successful
0929 20:23:11.944345 (+ 422us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":511,"thread_start_us":326,"threads_started":1}
I20260929 20:23:11.947029 24274 heartbeater.cc:347] Connected to a master server at 127.17.53.126:34513
I20260929 20:23:11.947335 24274 heartbeater.cc:464] Registering TS with master...
I20260929 20:23:11.948050 24274 heartbeater.cc:511] Master 127.17.53.126:34513 requested a full tablet report, sending...
I20260929 20:23:11.949469 23817 ts_manager.cc:194] Registered new tserver with Master: cd595f79664a402192976b9a664b71e7 (127.17.53.67:46647)
I20260929 20:23:11.950232 23817 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.17.53.67@KRBTEST.COM'} at 127.17.53.67:56901
I20260929 20:23:11.958400 17621 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.17.53.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.17.53.125@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.17.53.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713339161492-17621-0/krb5kdc/oryx_127.17.53.125.keytab.
Entry for principal oryx/127.17.53.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713339161492-17621-0/krb5kdc/oryx_127.17.53.125.keytab.
I20260929 20:23:12.007243 17621 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
/tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713339161492-17621-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713339161492-17621-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713339161492-17621-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713339161492-17621-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.53.125:35701
--webserver_interface=127.17.53.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713339161492-17621-0/krb5kdc/oryx_127.17.53.125.keytab
--principal=oryx/127.17.53.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.53.84:41905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.53.126:34513,127.17.53.125:35701
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713339161492-17621-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713339161492-17621-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713339161492-17621-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:23:12.337821 24279 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:23:12.338105 24279 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:23:12.338392 24279 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:23:12.345726 24279 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:23:12.345831 24279 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:23:12.345939 24279 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:23:12.346038 24279 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:23:12.346122 24279 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:23:12.356288 24279 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.53.84:41905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713339161492-17621-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713339161492-17621-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.17.53.126:34513,127.17.53.125:35701
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713339161492-17621-0/krb5kdc/oryx_127.17.53.125.keytab
--principal=oryx/127.17.53.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.53.125:35701
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713339161492-17621-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.17.53.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713339161492-17621-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
W20260929 20:23:12.356680 24279 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.17.53.126:34513,127.17.53.125:35701'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260929 20:23:12.357746 24279 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:23:12.359093 24279 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:23:12.371938 24287 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:23:12.372057 24285 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:23:12.372005 24284 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:23:12.373104 24279 server_base.cc:1034] running on GCE node
I20260929 20:23:12.373862 24279 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:23:12.375308 24279 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:23:12.376545 24279 hybrid_clock.cc:648] HybridClock initialized: now 1790713392376444 us; error 78 us; skew 500 ppm
I20260929 20:23:12.376878 24279 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:23:12 dist-test-slave-zbnc krb5kdc[23772](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713392, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.53.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:23:12.380333 24279 init.cc:377] Logged in from keytab as oryx/127.17.53.125@KRBTEST.COM (short username oryx)
I20260929 20:23:12.381806 24279 webserver.cc:460] Webserver started at http://127.17.53.125:37773/ using document root <none> and password file <none>
I20260929 20:23:12.382489 24279 fs_manager.cc:359] Metadata directory not provided
I20260929 20:23:12.382619 24279 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:23:12.383024 24279 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:23:12.385879 24279 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713339161492-17621-0/minicluster-data/master-1/data/instance:
uuid: "bbbb7fddd6f64d79bb894cf8c495922e"
format_stamp: "Formatted at 2026-09-29 20:23:12 on dist-test-slave-zbnc"
server_key: "19828165ce7e18dbd5e2fd2807b531ce"
server_key_iv: "43b6db9537c78e8b3973fed44bc62d73"
server_key_version: "encryptionkey@0"
I20260929 20:23:12.386629 24279 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713339161492-17621-0/minicluster-data/master-1/wal/instance:
uuid: "bbbb7fddd6f64d79bb894cf8c495922e"
format_stamp: "Formatted at 2026-09-29 20:23:12 on dist-test-slave-zbnc"
server_key: "19828165ce7e18dbd5e2fd2807b531ce"
server_key_iv: "43b6db9537c78e8b3973fed44bc62d73"
server_key_version: "encryptionkey@0"
I20260929 20:23:12.391692 24279 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260929 20:23:12.395088 24294 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:23:12.396394 24279 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 20:23:12.396566 24279 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713339161492-17621-0/minicluster-data/master-1/data,/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713339161492-17621-0/minicluster-data/master-1/wal
uuid: "bbbb7fddd6f64d79bb894cf8c495922e"
format_stamp: "Formatted at 2026-09-29 20:23:12 on dist-test-slave-zbnc"
server_key: "19828165ce7e18dbd5e2fd2807b531ce"
server_key_iv: "43b6db9537c78e8b3973fed44bc62d73"
server_key_version: "encryptionkey@0"
I20260929 20:23:12.396788 24279 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713339161492-17621-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713339161492-17621-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713339161492-17621-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:23:12.416584 24279 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:23:12.420523 24279 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:23:12.420799 24279 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:23:12.439951 24279 rpc_server.cc:307] RPC server started. Bound to: 127.17.53.125:35701
I20260929 20:23:12.439973 24345 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.53.125:35701 every 8 connection(s)
I20260929 20:23:12.441313 24279 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713339161492-17621-0/minicluster-data/master-1/data/info.pb
I20260929 20:23:12.444837 24346 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:23:12.447199 17621 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu as pid 24279
I20260929 20:23:12.454324 24346 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.53.126" port: 34513 } has no permanent_uuid. Determining permanent_uuid...
I20260929 20:23:12.458846 24140 heartbeater.cc:503] Master 127.17.53.126:34513 was elected leader, sending a full tablet report...
Sep 29 20:23:12 dist-test-slave-zbnc krb5kdc[23772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713389, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.17.53.125@KRBTEST.COM
Sep 29 20:23:12 dist-test-slave-zbnc krb5kdc[23772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713392, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.53.125@KRBTEST.COM for oryx/127.17.53.126@KRBTEST.COM
I20260929 20:23:12.471648 24349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:12.449437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48002 (local address 127.17.53.125:35701)
0929 20:23:12.450214 (+ 777us) server_negotiation.cc:206] Beginning negotiation
0929 20:23:12.450225 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 20:23:12.450266 (+ 41us) server_negotiation.cc:407] Connection header received
0929 20:23:12.451834 (+ 1568us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:23:12.451860 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:23:12.452251 (+ 391us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:23:12.452560 (+ 309us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:23:12.454440 (+ 1880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:12.455902 (+ 1462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:12.457497 (+ 1595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:12.458297 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:12.458457 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:12.463482 (+ 5025us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:23:12.463491 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:23:12.463505 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:23:12.463546 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:23:12.466730 (+ 3184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:23:12.467579 (+ 849us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:23:12.467588 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:23:12.467597 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:23:12.467810 (+ 213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:23:12.468236 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:23:12.468244 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:23:12.468246 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:23:12.468982 (+ 736us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:23:12.469298 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0929 20:23:12.469723 (+ 425us) server_negotiation.cc:299] Negotiation successful
0929 20:23:12.470187 (+ 464us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":567,"thread_start_us":371,"threads_started":1}
I20260929 20:23:12.474151 24350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:12.456940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59668 (local address 127.17.53.126:34513)
0929 20:23:12.457411 (+ 471us) server_negotiation.cc:206] Beginning negotiation
0929 20:23:12.457419 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 20:23:12.458004 (+ 585us) server_negotiation.cc:407] Connection header received
0929 20:23:12.458187 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:23:12.458195 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:23:12.458277 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:23:12.458394 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:23:12.460050 (+ 1656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:12.461326 (+ 1276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:12.463312 (+ 1986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:12.463936 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:12.464077 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:12.467683 (+ 3606us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:23:12.467690 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:23:12.467697 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:23:12.467738 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:23:12.471310 (+ 3572us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:23:12.472096 (+ 786us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:23:12.472101 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:23:12.472104 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:23:12.472229 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:23:12.472645 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:23:12.472649 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:23:12.472651 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:23:12.472934 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:23:12.473105 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0929 20:23:12.473570 (+ 465us) server_negotiation.cc:299] Negotiation successful
0929 20:23:12.473785 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":356,"thread_start_us":248,"threads_started":1}
I20260929 20:23:12.474210 24351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:12.456946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:34513 (local address 127.0.0.1:59668)
0929 20:23:12.457854 (+ 908us) negotiation.cc:107] Waiting for socket to connect
0929 20:23:12.457901 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0929 20:23:12.458055 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:23:12.458762 (+ 707us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:23:12.458769 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:23:12.459552 (+ 783us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:23:12.459881 (+ 329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:23:12.459893 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:23:12.461459 (+ 1566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:23:12.461463 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:23:12.463158 (+ 1695us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:23:12.463172 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:23:12.464141 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:23:12.464146 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:23:12.464385 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:12.465035 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:23:12.465086 (+ 51us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:23:12.467522 (+ 2436us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:23:12.471431 (+ 3909us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:23:12.471440 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:23:12.471445 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:23:12.471973 (+ 528us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:23:12.472338 (+ 365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:23:12.472344 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:23:12.472348 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:23:12.472480 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:23:12.473084 (+ 604us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:23:12.473091 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:23:12.473268 (+ 177us) client_negotiation.cc:769] Sending connection context
0929 20:23:12.473458 (+ 190us) client_negotiation.cc:240] Negotiation successful
0929 20:23:12.473779 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":664,"thread_start_us":519,"threads_started":1}
I20260929 20:23:12.475725 24346 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.53.125" port: 35701 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 20:23:12 dist-test-slave-zbnc krb5kdc[23772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713392, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.53.125@KRBTEST.COM for oryx/127.17.53.125@KRBTEST.COM
I20260929 20:23:12.487027 24351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:12.476376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.125:35701 (local address 127.0.0.1:48018)
0929 20:23:12.476582 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0929 20:23:12.476604 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 20:23:12.476678 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:23:12.476923 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:23:12.476926 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:23:12.477232 (+ 306us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:23:12.477353 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:23:12.477360 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:23:12.478268 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:23:12.478272 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:23:12.479223 (+ 951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:23:12.479235 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:23:12.479852 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:23:12.479855 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:23:12.479971 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:12.480471 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:23:12.480487 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:23:12.482359 (+ 1872us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:23:12.485122 (+ 2763us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:23:12.485126 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:23:12.485128 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:23:12.485475 (+ 347us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:23:12.485789 (+ 314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:23:12.485792 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:23:12.485794 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:23:12.485907 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:23:12.486326 (+ 419us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:23:12.486329 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:23:12.486442 (+ 113us) client_negotiation.cc:769] Sending connection context
0929 20:23:12.486544 (+ 102us) client_negotiation.cc:240] Negotiation successful
0929 20:23:12.486746 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57}
I20260929 20:23:12.487213 24349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:12.476504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48018 (local address 127.17.53.125:35701)
0929 20:23:12.476646 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0929 20:23:12.476650 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:23:12.476664 (+ 14us) server_negotiation.cc:407] Connection header received
0929 20:23:12.476765 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:23:12.476768 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:23:12.476818 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:23:12.476889 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:23:12.477460 (+ 571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:12.478140 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:12.479340 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:12.479723 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:12.479818 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:12.482502 (+ 2684us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:23:12.482506 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:23:12.482509 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:23:12.482535 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:23:12.485003 (+ 2468us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:23:12.485571 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:23:12.485575 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:23:12.485577 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:23:12.485693 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:23:12.485994 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:23:12.485997 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:23:12.485998 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:23:12.486234 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:23:12.486335 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0929 20:23:12.486625 (+ 290us) server_negotiation.cc:299] Negotiation successful
0929 20:23:12.486839 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":52,"server-negotiator.queue_time_us":51}
I20260929 20:23:12.491771 24346 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bbbb7fddd6f64d79bb894cf8c495922e: Bootstrap starting.
I20260929 20:23:12.494705 24346 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bbbb7fddd6f64d79bb894cf8c495922e: Neither blocks nor log segments found. Creating new log.
I20260929 20:23:12.495748 24346 log.cc:824] T 00000000000000000000000000000000 P bbbb7fddd6f64d79bb894cf8c495922e: Log is configured to *not* fsync() on all Append() calls
I20260929 20:23:12.498256 24346 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bbbb7fddd6f64d79bb894cf8c495922e: No bootstrap required, opened a new log
I20260929 20:23:12.505332 24346 raft_consensus.cc:348] T 00000000000000000000000000000000 P bbbb7fddd6f64d79bb894cf8c495922e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94c779fe1aa149a9add4b5b63f72b2df" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 34513 } } peers { permanent_uuid: "bbbb7fddd6f64d79bb894cf8c495922e" member_type: VOTER last_known_addr { host: "127.17.53.125" port: 35701 } }
I20260929 20:23:12.505643 24346 raft_consensus.cc:374] T 00000000000000000000000000000000 P bbbb7fddd6f64d79bb894cf8c495922e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:23:12.505704 24346 raft_consensus.cc:729] T 00000000000000000000000000000000 P bbbb7fddd6f64d79bb894cf8c495922e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbbb7fddd6f64d79bb894cf8c495922e, State: Initialized, Role: FOLLOWER
I20260929 20:23:12.506192 24346 consensus_queue.cc:260] T 00000000000000000000000000000000 P bbbb7fddd6f64d79bb894cf8c495922e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94c779fe1aa149a9add4b5b63f72b2df" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 34513 } } peers { permanent_uuid: "bbbb7fddd6f64d79bb894cf8c495922e" member_type: VOTER last_known_addr { host: "127.17.53.125" port: 35701 } }
I20260929 20:23:12.507045 24353 sys_catalog.cc:455] T 00000000000000000000000000000000 P bbbb7fddd6f64d79bb894cf8c495922e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94c779fe1aa149a9add4b5b63f72b2df" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 34513 } } peers { permanent_uuid: "bbbb7fddd6f64d79bb894cf8c495922e" member_type: VOTER last_known_addr { host: "127.17.53.125" port: 35701 } } }
I20260929 20:23:12.507381 24353 sys_catalog.cc:458] T 00000000000000000000000000000000 P bbbb7fddd6f64d79bb894cf8c495922e [sys.catalog]: This master's current role is: FOLLOWER
I20260929 20:23:12.508097 24346 sys_catalog.cc:564] T 00000000000000000000000000000000 P bbbb7fddd6f64d79bb894cf8c495922e [sys.catalog]: configured and running, proceeding with master startup.
W20260929 20:23:12.516408 24364 catalog_manager.cc:1558] T 00000000000000000000000000000000 P bbbb7fddd6f64d79bb894cf8c495922e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 20:23:12.516557 24364 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 20:23:12.519176 24279 master_runner.cc:428] Detected that this master bbbb7fddd6f64d79bb894cf8c495922e is joining an existing cluster
I20260929 20:23:12.519285 24279 master_runner.cc:432] Initiating AddMaster RPC to add 127.17.53.125:35701
I20260929 20:23:12.542325 24350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:12.523331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59682 (local address 127.17.53.126:34513)
0929 20:23:12.523714 (+ 383us) server_negotiation.cc:206] Beginning negotiation
0929 20:23:12.523720 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:23:12.524847 (+ 1127us) server_negotiation.cc:407] Connection header received
0929 20:23:12.525119 (+ 272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:23:12.525125 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:23:12.525220 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:23:12.525325 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:23:12.526990 (+ 1665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:12.528421 (+ 1431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:12.531201 (+ 2780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:12.532070 (+ 869us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:12.532183 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:12.535482 (+ 3299us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:23:12.535489 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:23:12.535494 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:23:12.535536 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:23:12.539077 (+ 3541us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:23:12.539898 (+ 821us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:23:12.539903 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:23:12.539907 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:23:12.540071 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:23:12.540531 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:23:12.540536 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:23:12.540539 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:23:12.540904 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:23:12.541051 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0929 20:23:12.541683 (+ 632us) server_negotiation.cc:299] Negotiation successful
0929 20:23:12.541927 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":257}
I20260929 20:23:12.543502 24371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:12.523467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:34513 (local address 127.0.0.1:59682)
0929 20:23:12.524745 (+ 1278us) negotiation.cc:107] Waiting for socket to connect
0929 20:23:12.524766 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 20:23:12.524961 (+ 195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:23:12.525364 (+ 403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:23:12.525368 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:23:12.526209 (+ 841us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:23:12.526756 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:23:12.526774 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:23:12.528650 (+ 1876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:23:12.528658 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:23:12.531006 (+ 2348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:23:12.531026 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:23:12.532557 (+ 1531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:23:12.532566 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:23:12.532837 (+ 271us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:12.533729 (+ 892us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:23:12.533761 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:23:12.535288 (+ 1527us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:23:12.539236 (+ 3948us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:23:12.539242 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:23:12.539246 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:23:12.539754 (+ 508us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:23:12.540225 (+ 471us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:23:12.540230 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:23:12.540233 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:23:12.540403 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:23:12.541141 (+ 738us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:23:12.541146 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:23:12.541365 (+ 219us) client_negotiation.cc:769] Sending connection context
0929 20:23:12.542612 (+ 1247us) client_negotiation.cc:240] Negotiation successful
0929 20:23:12.543032 (+ 420us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1130,"threads_started":1}
WARNING: no policy specified for oryx/127.17.53.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.17.53.124@KRBTEST.COM" created.
I20260929 20:23:12.560221 24349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:12.523517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48022 (local address 127.17.53.125:35701)
0929 20:23:12.523759 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0929 20:23:12.523767 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 20:23:12.543805 (+ 20038us) server_negotiation.cc:407] Connection header received
0929 20:23:12.544028 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:23:12.544033 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:23:12.544121 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:23:12.544188 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:23:12.545233 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:12.546385 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:12.548228 (+ 1843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:12.548947 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:12.549048 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:12.551546 (+ 2498us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:23:12.551556 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:23:12.551561 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:23:12.551596 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:23:12.555640 (+ 4044us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:23:12.557612 (+ 1972us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:23:12.557618 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:23:12.557621 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:23:12.557778 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:23:12.558195 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:23:12.558199 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:23:12.558203 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:23:12.558656 (+ 453us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:23:12.558845 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0929 20:23:12.559419 (+ 574us) server_negotiation.cc:299] Negotiation successful
0929 20:23:12.559720 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260929 20:23:12.561399 24371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:12.524027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.125:35701 (local address 127.0.0.1:48022)
0929 20:23:12.543720 (+ 19693us) negotiation.cc:107] Waiting for socket to connect
0929 20:23:12.543742 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 20:23:12.543897 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:23:12.544359 (+ 462us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:23:12.544364 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:23:12.544875 (+ 511us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:23:12.545089 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:23:12.545101 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:23:12.546605 (+ 1504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:23:12.546610 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:23:12.548075 (+ 1465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:23:12.548090 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:23:12.549346 (+ 1256us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:23:12.549352 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:23:12.549539 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:12.550204 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:23:12.550227 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:23:12.551344 (+ 1117us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:23:12.555781 (+ 4437us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:23:12.555789 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:23:12.555793 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:23:12.557488 (+ 1695us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:23:12.557908 (+ 420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:23:12.557912 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:23:12.557915 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:23:12.558088 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:23:12.558941 (+ 853us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:23:12.558953 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:23:12.559131 (+ 178us) client_negotiation.cc:769] Sending connection context
0929 20:23:12.560546 (+ 1415us) client_negotiation.cc:240] Negotiation successful
0929 20:23:12.560867 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19585,"thread_start_us":617,"threads_started":1}
Sep 29 20:23:12 dist-test-slave-zbnc krb5kdc[23772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713390, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.53.126@KRBTEST.COM for oryx/127.17.53.125@KRBTEST.COM
I20260929 20:23:12.586906 24349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:12.558245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48038 (local address 127.17.53.125:35701)
0929 20:23:12.560513 (+ 2268us) server_negotiation.cc:206] Beginning negotiation
0929 20:23:12.560519 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:23:12.567457 (+ 6938us) server_negotiation.cc:407] Connection header received
0929 20:23:12.567847 (+ 390us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:23:12.567854 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:23:12.567961 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:23:12.568070 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:23:12.569710 (+ 1640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:12.571009 (+ 1299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:12.572968 (+ 1959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:12.573784 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:12.573928 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:12.578155 (+ 4227us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:23:12.578164 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:23:12.578173 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:23:12.578217 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:23:12.583785 (+ 5568us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:23:12.584697 (+ 912us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:23:12.584701 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:23:12.584703 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:23:12.584829 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:23:12.585267 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:23:12.585277 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:23:12.585281 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:23:12.585597 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:23:12.585761 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0929 20:23:12.586223 (+ 462us) server_negotiation.cc:299] Negotiation successful
0929 20:23:12.586457 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2175,"thread_start_us":8305,"threads_started":1}
I20260929 20:23:12.586946 24373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:12.557886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.125:35701 (local address 127.0.0.1:48038)
0929 20:23:12.567310 (+ 9424us) negotiation.cc:107] Waiting for socket to connect
0929 20:23:12.567355 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0929 20:23:12.567606 (+ 251us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:23:12.568205 (+ 599us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:23:12.568214 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:23:12.569188 (+ 974us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:23:12.569525 (+ 337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:23:12.569540 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:23:12.571163 (+ 1623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:23:12.571169 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:23:12.572801 (+ 1632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:23:12.572816 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:23:12.573973 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:23:12.573980 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:23:12.574197 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:12.575003 (+ 806us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:23:12.575027 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:23:12.577934 (+ 2907us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:23:12.583935 (+ 6001us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:23:12.583951 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:23:12.583960 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:23:12.584562 (+ 602us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:23:12.584956 (+ 394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:23:12.584961 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:23:12.584965 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:23:12.585138 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:23:12.585753 (+ 615us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:23:12.585762 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:23:12.585945 (+ 183us) client_negotiation.cc:769] Sending connection context
0929 20:23:12.586139 (+ 194us) client_negotiation.cc:240] Negotiation successful
0929 20:23:12.586489 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9163,"thread_start_us":9032,"threads_started":1}
I20260929 20:23:12.589234 23817 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.17.53.125:35701: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "bbbb7fddd6f64d79bb894cf8c495922e"
member_type: NON_VOTER
last_known_addr {
host: "127.17.53.125"
port: 35701
}
attrs {
promote: true
}
}
dest_uuid: "94c779fe1aa149a9add4b5b63f72b2df"
cas_config_opid_index: -1
I20260929 20:23:12.590245 23817 consensus_queue.cc:237] T 00000000000000000000000000000000 P 94c779fe1aa149a9add4b5b63f72b2df [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "94c779fe1aa149a9add4b5b63f72b2df" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 34513 } } peers { permanent_uuid: "bbbb7fddd6f64d79bb894cf8c495922e" member_type: NON_VOTER last_known_addr { host: "127.17.53.125" port: 35701 } attrs { promote: true } }
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.17.53.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713339161492-17621-0/krb5kdc/oryx_127.17.53.124.keytab.
Entry for principal oryx/127.17.53.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713339161492-17621-0/krb5kdc/oryx_127.17.53.124.keytab.
I20260929 20:23:12.593035 17621 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
/tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713339161492-17621-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713339161492-17621-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713339161492-17621-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713339161492-17621-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.53.124:36147
--webserver_interface=127.17.53.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713339161492-17621-0/krb5kdc/oryx_127.17.53.124.keytab
--principal=oryx/127.17.53.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.53.84:41905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.53.126:34513,127.17.53.125:35701,127.17.53.124:36147
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713339161492-17621-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713339161492-17621-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713339161492-17621-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 20:23:12.597666 24378 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 94c779fe1aa149a9add4b5b63f72b2df [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER bbbb7fddd6f64d79bb894cf8c495922e (127.17.53.125) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "94c779fe1aa149a9add4b5b63f72b2df" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 34513 } } peers { permanent_uuid: "bbbb7fddd6f64d79bb894cf8c495922e" member_type: NON_VOTER last_known_addr { host: "127.17.53.125" port: 35701 } attrs { promote: true } } }
I20260929 20:23:12.598598 24321 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bbbb7fddd6f64d79bb894cf8c495922e [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:23:12.602065 24378 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.17.53.125:35701
I20260929 20:23:12.602429 24321 raft_consensus.cc:1261] T 00000000000000000000000000000000 P bbbb7fddd6f64d79bb894cf8c495922e [term 1 FOLLOWER]: Refusing update from remote peer 94c779fe1aa149a9add4b5b63f72b2df: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260929 20:23:12.602511 24378 sys_catalog.cc:455] T 00000000000000000000000000000000 P 94c779fe1aa149a9add4b5b63f72b2df [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "94c779fe1aa149a9add4b5b63f72b2df" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "94c779fe1aa149a9add4b5b63f72b2df" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 34513 } } peers { permanent_uuid: "bbbb7fddd6f64d79bb894cf8c495922e" member_type: NON_VOTER last_known_addr { host: "127.17.53.125" port: 35701 } attrs { promote: true } } }
I20260929 20:23:12.602833 24279 master.cc:537] Master@127.17.53.125:35701 shutting down...
I20260929 20:23:12.604080 24377 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 94c779fe1aa149a9add4b5b63f72b2df [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbbb7fddd6f64d79bb894cf8c495922e" member_type: NON_VOTER last_known_addr { host: "127.17.53.125" port: 35701 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:23:12.602717 24378 sys_catalog.cc:458] T 00000000000000000000000000000000 P 94c779fe1aa149a9add4b5b63f72b2df [sys.catalog]: This master's current role is: LEADER
W20260929 20:23:12.607614 23804 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
I20260929 20:23:12.616726 24279 raft_consensus.cc:2229] T 00000000000000000000000000000000 P bbbb7fddd6f64d79bb894cf8c495922e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 20:23:12.617242 24279 raft_consensus.cc:2258] T 00000000000000000000000000000000 P bbbb7fddd6f64d79bb894cf8c495922e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:23:12.617502 24279 tablet_replica.cc:330] T 00000000000000000000000000000000 P bbbb7fddd6f64d79bb894cf8c495922e: stopping tablet replica
W20260929 20:23:12.619903 23804 consensus_peers.cc:487] T 00000000000000000000000000000000 P 94c779fe1aa149a9add4b5b63f72b2df -> Peer bbbb7fddd6f64d79bb894cf8c495922e (127.17.53.125:35701): Couldn't send request to peer bbbb7fddd6f64d79bb894cf8c495922e. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260929 20:23:12.624132 24279 master.cc:552] Master@127.17.53.125:35701 shutdown complete.
I20260929 20:23:12.624200 24279 master_runner.cc:305] Clearing existing system tablet
I20260929 20:23:12.624532 24279 fs_manager.cc:359] Metadata directory not provided
I20260929 20:23:12.624596 24279 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:23:12.626730 24279 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.001s
I20260929 20:23:12.628228 24383 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:23:12.628531 24279 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713339161492-17621-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:23:12.628808 24279 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260929 20:23:12.628893 24279 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713339161492-17621-0/minicluster-data/master-1/data,/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713339161492-17621-0/minicluster-data/master-1/wal
uuid: "bbbb7fddd6f64d79bb894cf8c495922e"
format_stamp: "Formatted at 2026-09-29 20:23:12 on dist-test-slave-zbnc"
server_key: "19828165ce7e18dbd5e2fd2807b531ce"
server_key_iv: "43b6db9537c78e8b3973fed44bc62d73"
server_key_version: "encryptionkey@0"
I20260929 20:23:12.632550 24279 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P bbbb7fddd6f64d79bb894cf8c495922e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:23:12.637036 24279 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P bbbb7fddd6f64d79bb894cf8c495922e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260929 20:23:12.637226 24279 log.cc:1196] T 00000000000000000000000000000000 P bbbb7fddd6f64d79bb894cf8c495922e: Deleting WAL directory at /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713339161492-17621-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260929 20:23:12.637607 24279 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P bbbb7fddd6f64d79bb894cf8c495922e: Deleting consensus metadata
I20260929 20:23:12.637874 24279 master_runner.cc:315] Copying system tablet from 127.17.53.126:34513
I20260929 20:23:12.640532 24279 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P bbbb7fddd6f64d79bb894cf8c495922e: tablet copy: Beginning tablet copy session from remote peer at address 127.17.53.126:34513
I20260929 20:23:12.658252 24350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:12.641291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59696 (local address 127.17.53.126:34513)
0929 20:23:12.641501 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0929 20:23:12.641512 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 20:23:12.647428 (+ 5916us) server_negotiation.cc:407] Connection header received
0929 20:23:12.647643 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:23:12.647653 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:23:12.647722 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:23:12.647829 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:23:12.648606 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:12.649403 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:12.650641 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:12.651125 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:12.651268 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:12.653064 (+ 1796us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:23:12.653069 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:23:12.653072 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:23:12.653103 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:23:12.655743 (+ 2640us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:23:12.656342 (+ 599us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:23:12.656349 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:23:12.656353 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:23:12.656467 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:23:12.656842 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:23:12.656848 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:23:12.656851 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:23:12.657117 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:23:12.657275 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0929 20:23:12.657671 (+ 396us) server_negotiation.cc:299] Negotiation successful
0929 20:23:12.657868 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260929 20:23:12.658290 24389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:12.641291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:34513 (local address 127.0.0.1:59696)
0929 20:23:12.647289 (+ 5998us) negotiation.cc:107] Waiting for socket to connect
0929 20:23:12.647306 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 20:23:12.647497 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:23:12.647851 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:23:12.647855 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:23:12.648264 (+ 409us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:23:12.648456 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:23:12.648466 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:23:12.649553 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:23:12.649558 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:23:12.650494 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:23:12.650505 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:23:12.651281 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:23:12.651287 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:23:12.651456 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:12.652078 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:23:12.652095 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:23:12.652866 (+ 771us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:23:12.655867 (+ 3001us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:23:12.655874 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:23:12.655878 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:23:12.656226 (+ 348us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:23:12.656574 (+ 348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:23:12.656579 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:23:12.656583 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:23:12.656732 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:23:12.657250 (+ 518us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:23:12.657256 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:23:12.657424 (+ 168us) client_negotiation.cc:769] Sending connection context
0929 20:23:12.657582 (+ 158us) client_negotiation.cc:240] Negotiation successful
0929 20:23:12.657868 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5849,"thread_start_us":5735,"threads_started":1}
I20260929 20:23:12.659370 23837 tablet_copy_service.cc:140] P 94c779fe1aa149a9add4b5b63f72b2df: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer bbbb7fddd6f64d79bb894cf8c495922e ({username='oryx', principal='oryx/127.17.53.125@KRBTEST.COM'} at 127.0.0.1:59696)
I20260929 20:23:12.659631 23837 tablet_copy_service.cc:161] P 94c779fe1aa149a9add4b5b63f72b2df: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer bbbb7fddd6f64d79bb894cf8c495922e at {username='oryx', principal='oryx/127.17.53.125@KRBTEST.COM'} at 127.0.0.1:59696: session id = bbbb7fddd6f64d79bb894cf8c495922e-00000000000000000000000000000000
I20260929 20:23:12.662575 23837 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 94c779fe1aa149a9add4b5b63f72b2df: Tablet Copy: opened 0 blocks and 1 log segments
I20260929 20:23:12.664475 24279 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260929 20:23:12.669019 24279 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P bbbb7fddd6f64d79bb894cf8c495922e: tablet copy: Starting download of 0 data blocks...
I20260929 20:23:12.669312 24279 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P bbbb7fddd6f64d79bb894cf8c495922e: tablet copy: Starting download of 1 WAL segments...
I20260929 20:23:12.671746 24279 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P bbbb7fddd6f64d79bb894cf8c495922e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260929 20:23:12.674824 23837 tablet_copy_service.cc:342] P 94c779fe1aa149a9add4b5b63f72b2df: Request end of tablet copy session bbbb7fddd6f64d79bb894cf8c495922e-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.17.53.125@KRBTEST.COM'} at 127.0.0.1:59696
I20260929 20:23:12.675067 23837 tablet_copy_service.cc:433] P 94c779fe1aa149a9add4b5b63f72b2df: ending tablet copy session bbbb7fddd6f64d79bb894cf8c495922e-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer bbbb7fddd6f64d79bb894cf8c495922e
I20260929 20:23:12.685043 24279 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:23:12.687786 24390 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:23:12.698038 24279 server_base.cc:1034] running on GCE node
W20260929 20:23:12.698524 24393 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:23:12.698526 24391 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:23:12.699143 24279 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:23:12.699788 24279 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:23:12.701036 24279 hybrid_clock.cc:648] HybridClock initialized: now 1790713392700964 us; error 38 us; skew 500 ppm
I20260929 20:23:12.701300 24279 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:23:12 dist-test-slave-zbnc krb5kdc[23772](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713392, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.53.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:23:12.703364 24279 init.cc:377] Logged in from keytab as oryx/127.17.53.125@KRBTEST.COM (short username oryx)
I20260929 20:23:12.704790 24279 webserver.cc:460] Webserver started at http://127.17.53.125:36219/ using document root <none> and password file <none>
I20260929 20:23:12.705175 24279 fs_manager.cc:359] Metadata directory not provided
I20260929 20:23:12.705274 24279 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:23:12.708468 24279 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.003s
I20260929 20:23:12.710214 24400 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:23:12.710667 24279 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20260929 20:23:12.710825 24279 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713339161492-17621-0/minicluster-data/master-1/data,/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713339161492-17621-0/minicluster-data/master-1/wal
uuid: "bbbb7fddd6f64d79bb894cf8c495922e"
format_stamp: "Formatted at 2026-09-29 20:23:12 on dist-test-slave-zbnc"
server_key: "19828165ce7e18dbd5e2fd2807b531ce"
server_key_iv: "43b6db9537c78e8b3973fed44bc62d73"
server_key_version: "encryptionkey@0"
I20260929 20:23:12.711054 24279 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713339161492-17621-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713339161492-17621-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713339161492-17621-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:23:12.727253 24279 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:23:12.731215 24279 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:23:12.755805 24279 rpc_server.cc:307] RPC server started. Bound to: 127.17.53.125:35701
I20260929 20:23:12.755827 24451 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.53.125:35701 every 8 connection(s)
I20260929 20:23:12.756727 24279 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713339161492-17621-0/minicluster-data/master-1/data/info.pb
I20260929 20:23:12.763926 24452 sys_catalog.cc:263] Verifying existing consensus state
I20260929 20:23:12.766513 24452 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bbbb7fddd6f64d79bb894cf8c495922e: Bootstrap starting.
I20260929 20:23:12.779276 24452 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bbbb7fddd6f64d79bb894cf8c495922e: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 20:23:12.779718 24452 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bbbb7fddd6f64d79bb894cf8c495922e: Bootstrap complete.
I20260929 20:23:12.780529 24452 raft_consensus.cc:348] T 00000000000000000000000000000000 P bbbb7fddd6f64d79bb894cf8c495922e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "94c779fe1aa149a9add4b5b63f72b2df" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 34513 } } peers { permanent_uuid: "bbbb7fddd6f64d79bb894cf8c495922e" member_type: NON_VOTER last_known_addr { host: "127.17.53.125" port: 35701 } attrs { promote: true } }
I20260929 20:23:12.780752 24452 raft_consensus.cc:729] T 00000000000000000000000000000000 P bbbb7fddd6f64d79bb894cf8c495922e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: bbbb7fddd6f64d79bb894cf8c495922e, State: Initialized, Role: LEARNER
I20260929 20:23:12.780959 24452 consensus_queue.cc:260] T 00000000000000000000000000000000 P bbbb7fddd6f64d79bb894cf8c495922e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "94c779fe1aa149a9add4b5b63f72b2df" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 34513 } } peers { permanent_uuid: "bbbb7fddd6f64d79bb894cf8c495922e" member_type: NON_VOTER last_known_addr { host: "127.17.53.125" port: 35701 } attrs { promote: true } }
I20260929 20:23:12.781738 24455 sys_catalog.cc:455] T 00000000000000000000000000000000 P bbbb7fddd6f64d79bb894cf8c495922e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "94c779fe1aa149a9add4b5b63f72b2df" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 34513 } } peers { permanent_uuid: "bbbb7fddd6f64d79bb894cf8c495922e" member_type: NON_VOTER last_known_addr { host: "127.17.53.125" port: 35701 } attrs { promote: true } } }
I20260929 20:23:12.782054 24452 sys_catalog.cc:564] T 00000000000000000000000000000000 P bbbb7fddd6f64d79bb894cf8c495922e [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:23:12.782105 24455 sys_catalog.cc:458] T 00000000000000000000000000000000 P bbbb7fddd6f64d79bb894cf8c495922e [sys.catalog]: This master's current role is: LEARNER
I20260929 20:23:12.789714 24466 catalog_manager.cc:1259] Loaded cluster ID: 8dd81d8e76ec4d3d8e283d6964a8f856
I20260929 20:23:12.790082 24466 catalog_manager.cc:1552] T 00000000000000000000000000000000 P bbbb7fddd6f64d79bb894cf8c495922e: loading cluster ID for follower catalog manager: success
I20260929 20:23:12.792594 24466 catalog_manager.cc:1574] T 00000000000000000000000000000000 P bbbb7fddd6f64d79bb894cf8c495922e: acquiring CA information for follower catalog manager: success
I20260929 20:23:12.794425 24466 catalog_manager.cc:1602] T 00000000000000000000000000000000 P bbbb7fddd6f64d79bb894cf8c495922e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 20:23:12.952682 24274 heartbeater.cc:503] Master 127.17.53.126:34513 was elected leader, sending a full tablet report...
W20260929 20:23:13.027083 24381 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:23:13.027390 24381 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:23:13.027575 24381 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:23:13.035482 24381 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:23:13.035595 24381 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:23:13.035671 24381 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:23:13.035774 24381 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:23:13.035835 24381 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:23:13.046128 24381 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.53.84:41905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713339161492-17621-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713339161492-17621-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.17.53.126:34513,127.17.53.125:35701,127.17.53.124:36147
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713339161492-17621-0/krb5kdc/oryx_127.17.53.124.keytab
--principal=oryx/127.17.53.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.53.124:36147
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713339161492-17621-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.17.53.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713339161492-17621-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:23:13.047645 24381 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:23:13.048939 24381 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:23:13.064361 24472 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:23:13.064599 24474 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:23:13.064330 24471 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:23:13.065333 24381 server_base.cc:1034] running on GCE node
I20260929 20:23:13.066166 24381 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:23:13.067698 24381 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:23:13.068895 24381 hybrid_clock.cc:648] HybridClock initialized: now 1790713393068825 us; error 56 us; skew 500 ppm
I20260929 20:23:13.069285 24381 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:23:13 dist-test-slave-zbnc krb5kdc[23772](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713393, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.53.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:23:13.072464 24381 init.cc:377] Logged in from keytab as oryx/127.17.53.124@KRBTEST.COM (short username oryx)
I20260929 20:23:13.073804 24381 webserver.cc:460] Webserver started at http://127.17.53.124:34943/ using document root <none> and password file <none>
I20260929 20:23:13.074432 24381 fs_manager.cc:359] Metadata directory not provided
I20260929 20:23:13.074533 24381 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:23:13.074785 24381 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:23:13.077787 24381 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713339161492-17621-0/minicluster-data/master-2/data/instance:
uuid: "cde5a4bc3bab4d4489868351ecc30196"
format_stamp: "Formatted at 2026-09-29 20:23:13 on dist-test-slave-zbnc"
server_key: "f956cf17a77d8178139ded474b6cf400"
server_key_iv: "af9d052cfbd6bf249450a40e759e7c3d"
server_key_version: "encryptionkey@0"
I20260929 20:23:13.078472 24381 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713339161492-17621-0/minicluster-data/master-2/wal/instance:
uuid: "cde5a4bc3bab4d4489868351ecc30196"
format_stamp: "Formatted at 2026-09-29 20:23:13 on dist-test-slave-zbnc"
server_key: "f956cf17a77d8178139ded474b6cf400"
server_key_iv: "af9d052cfbd6bf249450a40e759e7c3d"
server_key_version: "encryptionkey@0"
I20260929 20:23:13.083791 24381 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260929 20:23:13.087575 24481 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:23:13.088982 24381 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 20:23:13.089140 24381 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713339161492-17621-0/minicluster-data/master-2/data,/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713339161492-17621-0/minicluster-data/master-2/wal
uuid: "cde5a4bc3bab4d4489868351ecc30196"
format_stamp: "Formatted at 2026-09-29 20:23:13 on dist-test-slave-zbnc"
server_key: "f956cf17a77d8178139ded474b6cf400"
server_key_iv: "af9d052cfbd6bf249450a40e759e7c3d"
server_key_version: "encryptionkey@0"
I20260929 20:23:13.089319 24381 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713339161492-17621-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713339161492-17621-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713339161492-17621-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:23:13.097270 24483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:13.090081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48046 (local address 127.17.53.125:35701)
0929 20:23:13.091099 (+ 1018us) server_negotiation.cc:206] Beginning negotiation
0929 20:23:13.091104 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:23:13.091120 (+ 16us) server_negotiation.cc:407] Connection header received
0929 20:23:13.091177 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:23:13.091180 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:23:13.091243 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:23:13.091336 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:23:13.091864 (+ 528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:13.092861 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:13.095302 (+ 2441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:13.096359 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:13.096457 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:13.096622 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0929 20:23:13.096922 (+ 300us) server_negotiation.cc:299] Negotiation successful
0929 20:23:13.097016 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":913,"thread_start_us":777,"threads_started":1}
I20260929 20:23:13.097345 24482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:13.089877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.125:35701 (local address 127.0.0.1:48046)
0929 20:23:13.090359 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0929 20:23:13.090388 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0929 20:23:13.090544 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:23:13.091410 (+ 866us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:23:13.091415 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:23:13.091449 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:23:13.091698 (+ 249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:23:13.091718 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:23:13.093008 (+ 1290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:23:13.093016 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:23:13.095093 (+ 2077us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:23:13.095112 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:23:13.096551 (+ 1439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:23:13.096558 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:23:13.096748 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:13.096799 (+ 51us) client_negotiation.cc:769] Sending connection context
0929 20:23:13.096889 (+ 90us) client_negotiation.cc:240] Negotiation successful
0929 20:23:13.097019 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":334,"thread_start_us":238,"threads_started":1}
I20260929 20:23:13.098351 24427 raft_consensus.cc:1203] T 00000000000000000000000000000000 P bbbb7fddd6f64d79bb894cf8c495922e [term 1 LEARNER]: Deduplicated request from leader. Original: 0.0->[1.1-1.5] Dedup: 1.5->[]
I20260929 20:23:13.098910 24455 sys_catalog.cc:455] T 00000000000000000000000000000000 P bbbb7fddd6f64d79bb894cf8c495922e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 94c779fe1aa149a9add4b5b63f72b2df. Latest consensus state: current_term: 1 leader_uuid: "94c779fe1aa149a9add4b5b63f72b2df" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "94c779fe1aa149a9add4b5b63f72b2df" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 34513 } } peers { permanent_uuid: "bbbb7fddd6f64d79bb894cf8c495922e" member_type: NON_VOTER last_known_addr { host: "127.17.53.125" port: 35701 } attrs { promote: true } } }
I20260929 20:23:13.099140 24455 sys_catalog.cc:458] T 00000000000000000000000000000000 P bbbb7fddd6f64d79bb894cf8c495922e [sys.catalog]: This master's current role is: LEARNER
I20260929 20:23:13.119474 24381 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:23:13.124007 24381 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:23:13.124300 24381 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:23:13.150607 24381 rpc_server.cc:307] RPC server started. Bound to: 127.17.53.124:36147
I20260929 20:23:13.150707 24534 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.53.124:36147 every 8 connection(s)
I20260929 20:23:13.152107 24381 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713339161492-17621-0/minicluster-data/master-2/data/info.pb
I20260929 20:23:13.159978 24535 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:23:13.161936 17621 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu as pid 24381
I20260929 20:23:13.171020 24535 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.53.126" port: 34513 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 20:23:13 dist-test-slave-zbnc krb5kdc[23772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713393, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.53.124@KRBTEST.COM for oryx/127.17.53.126@KRBTEST.COM
Sep 29 20:23:13 dist-test-slave-zbnc krb5kdc[23772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713389, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.17.53.124@KRBTEST.COM
I20260929 20:23:13.193150 24540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:13.174614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59712 (local address 127.17.53.126:34513)
0929 20:23:13.176098 (+ 1484us) server_negotiation.cc:206] Beginning negotiation
0929 20:23:13.176105 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:23:13.176124 (+ 19us) server_negotiation.cc:407] Connection header received
0929 20:23:13.176191 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:23:13.176196 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:23:13.176274 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:23:13.176394 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:23:13.178161 (+ 1767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:13.178992 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:13.180775 (+ 1783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:13.181554 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:13.181690 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:13.185756 (+ 4066us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:23:13.185763 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:23:13.185768 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:23:13.185806 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:23:13.189411 (+ 3605us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:23:13.190503 (+ 1092us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:23:13.190510 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:23:13.190514 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:23:13.190670 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:23:13.191198 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:23:13.191204 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:23:13.191210 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:23:13.191507 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:23:13.191659 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0929 20:23:13.192507 (+ 848us) server_negotiation.cc:299] Negotiation successful
0929 20:23:13.192712 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1353,"thread_start_us":1244,"threads_started":1}
I20260929 20:23:13.194962 24539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:13.174514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:34513 (local address 127.0.0.1:59712)
0929 20:23:13.175400 (+ 886us) negotiation.cc:107] Waiting for socket to connect
0929 20:23:13.175450 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0929 20:23:13.175611 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:23:13.176497 (+ 886us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:23:13.176508 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:23:13.177441 (+ 933us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:23:13.177973 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:23:13.177989 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:23:13.179151 (+ 1162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:23:13.179158 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:23:13.180616 (+ 1458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:23:13.180630 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:23:13.181782 (+ 1152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:23:13.181789 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:23:13.182127 (+ 338us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:13.183080 (+ 953us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:23:13.183110 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:23:13.185569 (+ 2459us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:23:13.189599 (+ 4030us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:23:13.189611 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:23:13.189623 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:23:13.190366 (+ 743us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:23:13.190784 (+ 418us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:23:13.190792 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:23:13.190799 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:23:13.191076 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:23:13.191642 (+ 566us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:23:13.191651 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:23:13.192031 (+ 380us) client_negotiation.cc:769] Sending connection context
0929 20:23:13.192412 (+ 381us) client_negotiation.cc:240] Negotiation successful
0929 20:23:13.192909 (+ 497us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":592,"mutex_wait_us":40,"spinlock_wait_cycles":19840,"thread_start_us":399,"threads_started":1}
I20260929 20:23:13.195022 24538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:13.164279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44942 (local address 127.17.53.124:36147)
0929 20:23:13.171858 (+ 7579us) server_negotiation.cc:206] Beginning negotiation
0929 20:23:13.171870 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 20:23:13.171915 (+ 45us) server_negotiation.cc:407] Connection header received
0929 20:23:13.173651 (+ 1736us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:23:13.173674 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:23:13.174148 (+ 474us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:23:13.176739 (+ 2591us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:23:13.177773 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:13.179923 (+ 2150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:13.181446 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:13.182290 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:13.182429 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:13.187990 (+ 5561us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:23:13.187999 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:23:13.188011 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:23:13.188039 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:23:13.191177 (+ 3138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:23:13.191983 (+ 806us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:23:13.191990 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:23:13.191994 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:23:13.192145 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:23:13.192585 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:23:13.192594 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:23:13.192600 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:23:13.193047 (+ 447us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:23:13.193378 (+ 331us) server_negotiation.cc:1035] Waiting for connection context
0929 20:23:13.194101 (+ 723us) server_negotiation.cc:299] Negotiation successful
0929 20:23:13.194323 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":24,"server-negotiator.queue_time_us":7346,"thread_start_us":7082,"threads_started":1}
I20260929 20:23:13.197314 24535 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.53.125" port: 35701 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 20:23:13 dist-test-slave-zbnc krb5kdc[23772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713393, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.53.124@KRBTEST.COM for oryx/127.17.53.125@KRBTEST.COM
I20260929 20:23:13.211726 24483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:13.198131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48056 (local address 127.17.53.125:35701)
0929 20:23:13.198312 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0929 20:23:13.198318 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:23:13.198385 (+ 67us) server_negotiation.cc:407] Connection header received
0929 20:23:13.198543 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:23:13.198547 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:23:13.198607 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:23:13.198685 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:23:13.199605 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:13.200295 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:13.202300 (+ 2005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:13.202708 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:13.202799 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:13.206288 (+ 3489us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:23:13.206293 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:23:13.206296 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:23:13.206326 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:23:13.208962 (+ 2636us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:23:13.209581 (+ 619us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:23:13.209587 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:23:13.209590 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:23:13.209739 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:23:13.210191 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:23:13.210195 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:23:13.210197 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:23:13.210438 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:23:13.210550 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0929 20:23:13.211213 (+ 663us) server_negotiation.cc:299] Negotiation successful
0929 20:23:13.211423 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260929 20:23:13.211911 24539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:13.198141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.125:35701 (local address 127.0.0.1:48056)
0929 20:23:13.198317 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0929 20:23:13.198335 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 20:23:13.198436 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:23:13.198726 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:23:13.198731 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:23:13.199242 (+ 511us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:23:13.199448 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:23:13.199462 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:23:13.200422 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:23:13.200430 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:23:13.202139 (+ 1709us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:23:13.202157 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:23:13.202909 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:23:13.202915 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:23:13.203125 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:13.203713 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:23:13.203738 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:23:13.206103 (+ 2365us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:23:13.209092 (+ 2989us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:23:13.209100 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:23:13.209106 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:23:13.209466 (+ 360us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:23:13.209888 (+ 422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:23:13.209894 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:23:13.209900 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:23:13.210079 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:23:13.210556 (+ 477us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:23:13.210563 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:23:13.210733 (+ 170us) client_negotiation.cc:769] Sending connection context
0929 20:23:13.211104 (+ 371us) client_negotiation.cc:240] Negotiation successful
0929 20:23:13.211505 (+ 401us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":63}
I20260929 20:23:13.213066 24535 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.53.124" port: 36147 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 20:23:13 dist-test-slave-zbnc krb5kdc[23772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713393, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.53.124@KRBTEST.COM for oryx/127.17.53.124@KRBTEST.COM
I20260929 20:23:13.228286 24539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:13.213964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.124:36147 (local address 127.0.0.1:44954)
0929 20:23:13.214131 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0929 20:23:13.214152 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 20:23:13.214274 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:23:13.214547 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:23:13.214555 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:23:13.215057 (+ 502us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:23:13.215270 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:23:13.215283 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:23:13.216256 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:23:13.216263 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:23:13.217817 (+ 1554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:23:13.217833 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:23:13.218499 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:23:13.218506 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:23:13.218746 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:13.219356 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:23:13.219380 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:23:13.222067 (+ 2687us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:23:13.226002 (+ 3935us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:23:13.226006 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:23:13.226008 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:23:13.226354 (+ 346us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:23:13.226745 (+ 391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:23:13.226748 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:23:13.226751 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:23:13.226947 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:23:13.227565 (+ 618us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:23:13.227568 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:23:13.227679 (+ 111us) client_negotiation.cc:769] Sending connection context
0929 20:23:13.227792 (+ 113us) client_negotiation.cc:240] Negotiation successful
0929 20:23:13.227959 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260929 20:23:13.228605 24538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:13.213900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44954 (local address 127.17.53.124:36147)
0929 20:23:13.214070 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0929 20:23:13.214075 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:23:13.214228 (+ 153us) server_negotiation.cc:407] Connection header received
0929 20:23:13.214372 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:23:13.214375 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:23:13.214425 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:23:13.214495 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:23:13.215403 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:13.216136 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:13.217952 (+ 1816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:13.218356 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:13.218452 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:13.222312 (+ 3860us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:23:13.222321 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:23:13.222327 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:23:13.222368 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:23:13.225852 (+ 3484us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:23:13.226462 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:23:13.226469 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:23:13.226474 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:23:13.226621 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:23:13.227045 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:23:13.227051 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:23:13.227056 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:23:13.227445 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:23:13.227590 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0929 20:23:13.227932 (+ 342us) server_negotiation.cc:299] Negotiation successful
0929 20:23:13.228191 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 20:23:13.235548 24535 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cde5a4bc3bab4d4489868351ecc30196: Bootstrap starting.
I20260929 20:23:13.239815 24535 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cde5a4bc3bab4d4489868351ecc30196: Neither blocks nor log segments found. Creating new log.
I20260929 20:23:13.240797 24535 log.cc:824] T 00000000000000000000000000000000 P cde5a4bc3bab4d4489868351ecc30196: Log is configured to *not* fsync() on all Append() calls
I20260929 20:23:13.243382 24535 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cde5a4bc3bab4d4489868351ecc30196: No bootstrap required, opened a new log
I20260929 20:23:13.250415 24535 raft_consensus.cc:348] T 00000000000000000000000000000000 P cde5a4bc3bab4d4489868351ecc30196 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94c779fe1aa149a9add4b5b63f72b2df" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 34513 } } peers { permanent_uuid: "bbbb7fddd6f64d79bb894cf8c495922e" member_type: VOTER last_known_addr { host: "127.17.53.125" port: 35701 } } peers { permanent_uuid: "cde5a4bc3bab4d4489868351ecc30196" member_type: VOTER last_known_addr { host: "127.17.53.124" port: 36147 } }
I20260929 20:23:13.250748 24535 raft_consensus.cc:374] T 00000000000000000000000000000000 P cde5a4bc3bab4d4489868351ecc30196 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:23:13.250837 24535 raft_consensus.cc:729] T 00000000000000000000000000000000 P cde5a4bc3bab4d4489868351ecc30196 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cde5a4bc3bab4d4489868351ecc30196, State: Initialized, Role: FOLLOWER
I20260929 20:23:13.251400 24535 consensus_queue.cc:260] T 00000000000000000000000000000000 P cde5a4bc3bab4d4489868351ecc30196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94c779fe1aa149a9add4b5b63f72b2df" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 34513 } } peers { permanent_uuid: "bbbb7fddd6f64d79bb894cf8c495922e" member_type: VOTER last_known_addr { host: "127.17.53.125" port: 35701 } } peers { permanent_uuid: "cde5a4bc3bab4d4489868351ecc30196" member_type: VOTER last_known_addr { host: "127.17.53.124" port: 36147 } }
I20260929 20:23:13.252140 24542 sys_catalog.cc:455] T 00000000000000000000000000000000 P cde5a4bc3bab4d4489868351ecc30196 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "94c779fe1aa149a9add4b5b63f72b2df" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 34513 } } peers { permanent_uuid: "bbbb7fddd6f64d79bb894cf8c495922e" member_type: VOTER last_known_addr { host: "127.17.53.125" port: 35701 } } peers { permanent_uuid: "cde5a4bc3bab4d4489868351ecc30196" member_type: VOTER last_known_addr { host: "127.17.53.124" port: 36147 } } }
I20260929 20:23:13.252508 24542 sys_catalog.cc:458] T 00000000000000000000000000000000 P cde5a4bc3bab4d4489868351ecc30196 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 20:23:13.253090 24535 sys_catalog.cc:564] T 00000000000000000000000000000000 P cde5a4bc3bab4d4489868351ecc30196 [sys.catalog]: configured and running, proceeding with master startup.
W20260929 20:23:13.261615 24553 catalog_manager.cc:1558] T 00000000000000000000000000000000 P cde5a4bc3bab4d4489868351ecc30196: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 20:23:13.261751 24553 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 20:23:13.265888 24381 master_runner.cc:428] Detected that this master cde5a4bc3bab4d4489868351ecc30196 is joining an existing cluster
I20260929 20:23:13.266010 24381 master_runner.cc:432] Initiating AddMaster RPC to add 127.17.53.124:36147
I20260929 20:23:13.291760 24538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:13.270829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44970 (local address 127.17.53.124:36147)
0929 20:23:13.271051 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0929 20:23:13.271057 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:23:13.271710 (+ 653us) server_negotiation.cc:407] Connection header received
0929 20:23:13.271879 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:23:13.271884 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:23:13.271961 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:23:13.272054 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:23:13.275120 (+ 3066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:13.276229 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:13.279842 (+ 3613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:13.280550 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:13.280643 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:13.285235 (+ 4592us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:23:13.285241 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:23:13.285247 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:23:13.285279 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:23:13.289048 (+ 3769us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:23:13.289778 (+ 730us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:23:13.289783 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:23:13.289787 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:23:13.289911 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:23:13.290238 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:23:13.290243 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:23:13.290247 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:23:13.290572 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:23:13.290719 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0929 20:23:13.291176 (+ 457us) server_negotiation.cc:299] Negotiation successful
0929 20:23:13.291374 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260929 20:23:13.292999 24559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:13.270414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.124:36147 (local address 127.0.0.1:44970)
0929 20:23:13.271624 (+ 1210us) negotiation.cc:107] Waiting for socket to connect
0929 20:23:13.271643 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 20:23:13.271767 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:23:13.272546 (+ 779us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:23:13.272551 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:23:13.273321 (+ 770us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:23:13.273697 (+ 376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:23:13.273709 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:23:13.276445 (+ 2736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:23:13.276450 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:23:13.277986 (+ 1536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:23:13.278000 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:23:13.280974 (+ 2974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:23:13.280980 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:23:13.281141 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:13.282479 (+ 1338us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:23:13.282502 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:23:13.284779 (+ 2277us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:23:13.289194 (+ 4415us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:23:13.289199 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:23:13.289203 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:23:13.289677 (+ 474us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:23:13.290016 (+ 339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:23:13.290020 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:23:13.290023 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:23:13.290152 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:23:13.290776 (+ 624us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:23:13.290781 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:23:13.290954 (+ 173us) client_negotiation.cc:769] Sending connection context
0929 20:23:13.292269 (+ 1315us) client_negotiation.cc:240] Negotiation successful
0929 20:23:13.292560 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1069,"thread_start_us":925,"threads_started":1}
I20260929 20:23:13.301602 17621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu pid 23878
I20260929 20:23:13.308483 24540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:13.270198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59724 (local address 127.17.53.126:34513)
0929 20:23:13.270385 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0929 20:23:13.270391 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:23:13.285635 (+ 15244us) server_negotiation.cc:407] Connection header received
0929 20:23:13.285936 (+ 301us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:23:13.285944 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:23:13.286039 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:23:13.286124 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:23:13.294432 (+ 8308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:13.295580 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:13.297969 (+ 2389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:13.298666 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:13.298801 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:13.301533 (+ 2732us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:23:13.301540 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:23:13.301545 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:23:13.301584 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:23:13.305272 (+ 3688us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:23:13.306120 (+ 848us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:23:13.306126 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:23:13.306130 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:23:13.306295 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:23:13.306792 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:23:13.306797 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:23:13.306800 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:23:13.307161 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:23:13.307308 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0929 20:23:13.307917 (+ 609us) server_negotiation.cc:299] Negotiation successful
0929 20:23:13.308148 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260929 20:23:13.309811 24560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:13.270766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:34513 (local address 127.0.0.1:59724)
0929 20:23:13.285532 (+ 14766us) negotiation.cc:107] Waiting for socket to connect
0929 20:23:13.285551 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 20:23:13.285772 (+ 221us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:23:13.286356 (+ 584us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:23:13.286360 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:23:13.293893 (+ 7533us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:23:13.294257 (+ 364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:23:13.294268 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:23:13.295785 (+ 1517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:23:13.295791 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:23:13.297810 (+ 2019us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:23:13.297825 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:23:13.299081 (+ 1256us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:23:13.299089 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:23:13.299270 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:13.299992 (+ 722us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:23:13.300019 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:23:13.301306 (+ 1287us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:23:13.305456 (+ 4150us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:23:13.305462 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:23:13.305465 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:23:13.305981 (+ 516us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:23:13.306443 (+ 462us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:23:13.306450 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:23:13.306454 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:23:13.306653 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:23:13.307393 (+ 740us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:23:13.307402 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:23:13.307610 (+ 208us) client_negotiation.cc:769] Sending connection context
0929 20:23:13.309009 (+ 1399us) client_negotiation.cc:240] Negotiation successful
0929 20:23:13.309335 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14595,"mutex_wait_us":349,"thread_start_us":14065,"threads_started":1}
I20260929 20:23:13.310286 24483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:13.272470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48064 (local address 127.17.53.125:35701)
0929 20:23:13.272635 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0929 20:23:13.272642 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:23:13.293578 (+ 20936us) server_negotiation.cc:407] Connection header received
0929 20:23:13.293790 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:23:13.293796 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:23:13.293876 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:23:13.293944 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:23:13.295150 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:13.296193 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:13.298070 (+ 1877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:13.298916 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:13.299036 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:13.301771 (+ 2735us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:23:13.301778 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:23:13.301782 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:23:13.301824 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:23:13.306229 (+ 4405us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:23:13.307400 (+ 1171us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:23:13.307408 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:23:13.307414 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:23:13.307628 (+ 214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:23:13.308169 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:23:13.308177 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:23:13.308183 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:23:13.308602 (+ 419us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:23:13.308762 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0929 20:23:13.309517 (+ 755us) server_negotiation.cc:299] Negotiation successful
0929 20:23:13.309842 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 20:23:13.312112 24559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:13.273009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.125:35701 (local address 127.0.0.1:48064)
0929 20:23:13.293468 (+ 20459us) negotiation.cc:107] Waiting for socket to connect
0929 20:23:13.293486 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 20:23:13.293666 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:23:13.294172 (+ 506us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:23:13.294177 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:23:13.294723 (+ 546us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:23:13.294981 (+ 258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:23:13.294992 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:23:13.296379 (+ 1387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:23:13.296384 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:23:13.297902 (+ 1518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:23:13.297914 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:23:13.299272 (+ 1358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:23:13.299277 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:23:13.299426 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:13.300165 (+ 739us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:23:13.300191 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:23:13.301561 (+ 1370us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:23:13.306427 (+ 4866us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:23:13.306435 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:23:13.306441 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:23:13.307226 (+ 785us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:23:13.307813 (+ 587us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:23:13.307820 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:23:13.307826 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:23:13.308024 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:23:13.308859 (+ 835us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:23:13.308866 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:23:13.309104 (+ 238us) client_negotiation.cc:769] Sending connection context
0929 20:23:13.310905 (+ 1801us) client_negotiation.cc:240] Negotiation successful
0929 20:23:13.311480 (+ 575us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20366,"mutex_wait_us":1,"thread_start_us":24526,"threads_started":1}
Sep 29 20:23:13 dist-test-slave-zbnc krb5kdc[23772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713390, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.53.126@KRBTEST.COM for oryx/127.17.53.124@KRBTEST.COM
Sep 29 20:23:13 dist-test-slave-zbnc krb5kdc[23772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713389, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.17.53.65@KRBTEST.COM
I20260929 20:23:13.332388 24482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:13.316037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.124:36147 (local address 127.0.0.1:44986)
0929 20:23:13.316206 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0929 20:23:13.316225 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 20:23:13.316332 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:23:13.316927 (+ 595us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:23:13.316933 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:23:13.317377 (+ 444us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:23:13.317568 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:23:13.317580 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:23:13.319105 (+ 1525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:23:13.319111 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:23:13.320505 (+ 1394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:23:13.320519 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:23:13.321745 (+ 1226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:23:13.321753 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:23:13.321938 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:13.322612 (+ 674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:23:13.322636 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:23:13.325903 (+ 3267us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:23:13.329771 (+ 3868us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:23:13.329776 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:23:13.329781 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:23:13.330221 (+ 440us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:23:13.330606 (+ 385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:23:13.330610 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:23:13.330614 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:23:13.330785 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:23:13.331450 (+ 665us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:23:13.331455 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:23:13.331639 (+ 184us) client_negotiation.cc:769] Sending connection context
0929 20:23:13.331808 (+ 169us) client_negotiation.cc:240] Negotiation successful
0929 20:23:13.332091 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260929 20:23:13.333194 24538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:13.316099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44986 (local address 127.17.53.124:36147)
0929 20:23:13.316262 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0929 20:23:13.316266 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:23:13.316280 (+ 14us) server_negotiation.cc:407] Connection header received
0929 20:23:13.316524 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:23:13.316530 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:23:13.316621 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:23:13.316726 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:23:13.317753 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:13.318940 (+ 1187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:13.320712 (+ 1772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:13.321407 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:13.321524 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:13.326129 (+ 4605us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:23:13.326135 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:23:13.326140 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:23:13.326184 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:23:13.329637 (+ 3453us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:23:13.330349 (+ 712us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:23:13.330355 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:23:13.330358 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:23:13.330488 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:23:13.330927 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:23:13.330933 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:23:13.330941 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:23:13.331222 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:23:13.331411 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0929 20:23:13.332774 (+ 1363us) server_negotiation.cc:299] Negotiation successful
0929 20:23:13.332953 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 20:23:13.334520 23817 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.17.53.124:36147: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "cde5a4bc3bab4d4489868351ecc30196"
member_type: NON_VOTER
last_known_addr {
host: "127.17.53.124"
port: 36147
}
attrs {
promote: true
}
}
dest_uuid: "94c779fe1aa149a9add4b5b63f72b2df"
cas_config_opid_index: 5
I20260929 20:23:13.335431 23817 consensus_queue.cc:237] T 00000000000000000000000000000000 P 94c779fe1aa149a9add4b5b63f72b2df [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "94c779fe1aa149a9add4b5b63f72b2df" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 34513 } } peers { permanent_uuid: "bbbb7fddd6f64d79bb894cf8c495922e" member_type: NON_VOTER last_known_addr { host: "127.17.53.125" port: 35701 } attrs { promote: true } } peers { permanent_uuid: "cde5a4bc3bab4d4489868351ecc30196" member_type: NON_VOTER last_known_addr { host: "127.17.53.124" port: 36147 } attrs { promote: true } }
I20260929 20:23:13.337368 24378 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 94c779fe1aa149a9add4b5b63f72b2df [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER cde5a4bc3bab4d4489868351ecc30196 (127.17.53.124) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "94c779fe1aa149a9add4b5b63f72b2df" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 34513 } } peers { permanent_uuid: "bbbb7fddd6f64d79bb894cf8c495922e" member_type: NON_VOTER last_known_addr { host: "127.17.53.125" port: 35701 } attrs { promote: true } } peers { permanent_uuid: "cde5a4bc3bab4d4489868351ecc30196" member_type: NON_VOTER last_known_addr { host: "127.17.53.124" port: 36147 } attrs { promote: true } } }
I20260929 20:23:13.338922 24562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:13.303089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45880 (local address 127.17.53.65:46815)
0929 20:23:13.318601 (+ 15512us) server_negotiation.cc:206] Beginning negotiation
0929 20:23:13.318608 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:23:13.318634 (+ 26us) server_negotiation.cc:407] Connection header received
0929 20:23:13.318686 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:23:13.318692 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:23:13.318997 (+ 305us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:23:13.319287 (+ 290us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:23:13.320636 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:13.321776 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:13.323770 (+ 1994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:13.324569 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:13.324698 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:13.330549 (+ 5851us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:23:13.330566 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:23:13.330587 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:23:13.330629 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:23:13.334649 (+ 4020us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:23:13.335603 (+ 954us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:23:13.335613 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:23:13.335616 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:23:13.335784 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:23:13.336282 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:23:13.336287 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:23:13.336290 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:23:13.336746 (+ 456us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:23:13.338028 (+ 1282us) server_negotiation.cc:1035] Waiting for connection context
0929 20:23:13.338218 (+ 190us) server_negotiation.cc:299] Negotiation successful
0929 20:23:13.338496 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15369,"thread_start_us":15813,"threads_started":1}
I20260929 20:23:13.339938 24427 raft_consensus.cc:1261] T 00000000000000000000000000000000 P bbbb7fddd6f64d79bb894cf8c495922e [term 1 LEARNER]: Refusing update from remote peer 94c779fe1aa149a9add4b5b63f72b2df: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260929 20:23:13.340826 24510 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cde5a4bc3bab4d4489868351ecc30196 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:23:13.343248 24563 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 94c779fe1aa149a9add4b5b63f72b2df [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbbb7fddd6f64d79bb894cf8c495922e" member_type: NON_VOTER last_known_addr { host: "127.17.53.125" port: 35701 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260929 20:23:13.343681 24510 raft_consensus.cc:1261] T 00000000000000000000000000000000 P cde5a4bc3bab4d4489868351ecc30196 [term 1 FOLLOWER]: Refusing update from remote peer 94c779fe1aa149a9add4b5b63f72b2df: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260929 20:23:13.345484 24378 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.17.53.124:36147
I20260929 20:23:13.347570 24001 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:23:13.347856 24381 master.cc:537] Master@127.17.53.124:36147 shutting down...
I20260929 20:23:13.347821 24378 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 94c779fe1aa149a9add4b5b63f72b2df [LEADER]: Connected to new peer: Peer: permanent_uuid: "cde5a4bc3bab4d4489868351ecc30196" member_type: NON_VOTER last_known_addr { host: "127.17.53.124" port: 36147 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:23:13.352703 24427 raft_consensus.cc:2932] T 00000000000000000000000000000000 P bbbb7fddd6f64d79bb894cf8c495922e [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER cde5a4bc3bab4d4489868351ecc30196 (127.17.53.124) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "94c779fe1aa149a9add4b5b63f72b2df" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 34513 } } peers { permanent_uuid: "bbbb7fddd6f64d79bb894cf8c495922e" member_type: NON_VOTER last_known_addr { host: "127.17.53.125" port: 35701 } attrs { promote: true } } peers { permanent_uuid: "cde5a4bc3bab4d4489868351ecc30196" member_type: NON_VOTER last_known_addr { host: "127.17.53.124" port: 36147 } attrs { promote: true } } }
I20260929 20:23:13.357407 24455 sys_catalog.cc:455] T 00000000000000000000000000000000 P bbbb7fddd6f64d79bb894cf8c495922e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "94c779fe1aa149a9add4b5b63f72b2df" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "94c779fe1aa149a9add4b5b63f72b2df" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 34513 } } peers { permanent_uuid: "bbbb7fddd6f64d79bb894cf8c495922e" member_type: NON_VOTER last_known_addr { host: "127.17.53.125" port: 35701 } attrs { promote: true } } peers { permanent_uuid: "cde5a4bc3bab4d4489868351ecc30196" member_type: NON_VOTER last_known_addr { host: "127.17.53.124" port: 36147 } attrs { promote: true } } }
I20260929 20:23:13.357679 24455 sys_catalog.cc:458] T 00000000000000000000000000000000 P bbbb7fddd6f64d79bb894cf8c495922e [sys.catalog]: This master's current role is: LEARNER
I20260929 20:23:13.351229 24563 sys_catalog.cc:455] T 00000000000000000000000000000000 P 94c779fe1aa149a9add4b5b63f72b2df [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "94c779fe1aa149a9add4b5b63f72b2df" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "94c779fe1aa149a9add4b5b63f72b2df" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 34513 } } peers { permanent_uuid: "bbbb7fddd6f64d79bb894cf8c495922e" member_type: NON_VOTER last_known_addr { host: "127.17.53.125" port: 35701 } attrs { promote: true } } peers { permanent_uuid: "cde5a4bc3bab4d4489868351ecc30196" member_type: NON_VOTER last_known_addr { host: "127.17.53.124" port: 36147 } attrs { promote: true } } }
I20260929 20:23:13.358309 24563 sys_catalog.cc:458] T 00000000000000000000000000000000 P 94c779fe1aa149a9add4b5b63f72b2df [sys.catalog]: This master's current role is: LEADER
I20260929 20:23:13.361119 24564 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 94c779fe1aa149a9add4b5b63f72b2df: attempting to promote NON_VOTER bbbb7fddd6f64d79bb894cf8c495922e to VOTER
I20260929 20:23:13.361907 24564 consensus_queue.cc:237] T 00000000000000000000000000000000 P 94c779fe1aa149a9add4b5b63f72b2df [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "94c779fe1aa149a9add4b5b63f72b2df" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 34513 } } peers { permanent_uuid: "bbbb7fddd6f64d79bb894cf8c495922e" member_type: VOTER last_known_addr { host: "127.17.53.125" port: 35701 } attrs { promote: false } } peers { permanent_uuid: "cde5a4bc3bab4d4489868351ecc30196" member_type: NON_VOTER last_known_addr { host: "127.17.53.124" port: 36147 } attrs { promote: true } }
W20260929 20:23:13.366454 23803 consensus_peers.cc:487] T 00000000000000000000000000000000 P 94c779fe1aa149a9add4b5b63f72b2df -> Peer cde5a4bc3bab4d4489868351ecc30196 (127.17.53.124:36147): Couldn't send request to peer cde5a4bc3bab4d4489868351ecc30196. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260929 20:23:13.367816 24427 raft_consensus.cc:1261] T 00000000000000000000000000000000 P bbbb7fddd6f64d79bb894cf8c495922e [term 1 LEARNER]: Refusing update from remote peer 94c779fe1aa149a9add4b5b63f72b2df: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260929 20:23:13.368537 24378 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 94c779fe1aa149a9add4b5b63f72b2df [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbbb7fddd6f64d79bb894cf8c495922e" member_type: VOTER last_known_addr { host: "127.17.53.125" port: 35701 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260929 20:23:13.371356 24569 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 94c779fe1aa149a9add4b5b63f72b2df [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, bbbb7fddd6f64d79bb894cf8c495922e (127.17.53.125) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "94c779fe1aa149a9add4b5b63f72b2df" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 34513 } } peers { permanent_uuid: "bbbb7fddd6f64d79bb894cf8c495922e" member_type: VOTER last_known_addr { host: "127.17.53.125" port: 35701 } attrs { promote: false } } peers { permanent_uuid: "cde5a4bc3bab4d4489868351ecc30196" member_type: NON_VOTER last_known_addr { host: "127.17.53.124" port: 36147 } attrs { promote: true } } }
I20260929 20:23:13.371845 24427 raft_consensus.cc:2932] T 00000000000000000000000000000000 P bbbb7fddd6f64d79bb894cf8c495922e [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, bbbb7fddd6f64d79bb894cf8c495922e (127.17.53.125) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "94c779fe1aa149a9add4b5b63f72b2df" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 34513 } } peers { permanent_uuid: "bbbb7fddd6f64d79bb894cf8c495922e" member_type: VOTER last_known_addr { host: "127.17.53.125" port: 35701 } attrs { promote: false } } peers { permanent_uuid: "cde5a4bc3bab4d4489868351ecc30196" member_type: NON_VOTER last_known_addr { host: "127.17.53.124" port: 36147 } attrs { promote: true } } }
I20260929 20:23:13.376009 24378 sys_catalog.cc:455] T 00000000000000000000000000000000 P 94c779fe1aa149a9add4b5b63f72b2df [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "94c779fe1aa149a9add4b5b63f72b2df" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "94c779fe1aa149a9add4b5b63f72b2df" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 34513 } } peers { permanent_uuid: "bbbb7fddd6f64d79bb894cf8c495922e" member_type: VOTER last_known_addr { host: "127.17.53.125" port: 35701 } attrs { promote: false } } peers { permanent_uuid: "cde5a4bc3bab4d4489868351ecc30196" member_type: NON_VOTER last_known_addr { host: "127.17.53.124" port: 36147 } attrs { promote: true } } }
I20260929 20:23:13.376091 24455 sys_catalog.cc:455] T 00000000000000000000000000000000 P bbbb7fddd6f64d79bb894cf8c495922e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "94c779fe1aa149a9add4b5b63f72b2df" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "94c779fe1aa149a9add4b5b63f72b2df" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 34513 } } peers { permanent_uuid: "bbbb7fddd6f64d79bb894cf8c495922e" member_type: VOTER last_known_addr { host: "127.17.53.125" port: 35701 } attrs { promote: false } } peers { permanent_uuid: "cde5a4bc3bab4d4489868351ecc30196" member_type: NON_VOTER last_known_addr { host: "127.17.53.124" port: 36147 } attrs { promote: true } } }
I20260929 20:23:13.376308 24378 sys_catalog.cc:458] T 00000000000000000000000000000000 P 94c779fe1aa149a9add4b5b63f72b2df [sys.catalog]: This master's current role is: LEADER
I20260929 20:23:13.376344 24455 sys_catalog.cc:458] T 00000000000000000000000000000000 P bbbb7fddd6f64d79bb894cf8c495922e [sys.catalog]: This master's current role is: FOLLOWER
I20260929 20:23:13.376550 24378 sys_catalog.cc:455] T 00000000000000000000000000000000 P 94c779fe1aa149a9add4b5b63f72b2df [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "94c779fe1aa149a9add4b5b63f72b2df" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "94c779fe1aa149a9add4b5b63f72b2df" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 34513 } } peers { permanent_uuid: "bbbb7fddd6f64d79bb894cf8c495922e" member_type: VOTER last_known_addr { host: "127.17.53.125" port: 35701 } attrs { promote: false } } peers { permanent_uuid: "cde5a4bc3bab4d4489868351ecc30196" member_type: NON_VOTER last_known_addr { host: "127.17.53.124" port: 36147 } attrs { promote: true } } }
I20260929 20:23:13.376770 24378 sys_catalog.cc:458] T 00000000000000000000000000000000 P 94c779fe1aa149a9add4b5b63f72b2df [sys.catalog]: This master's current role is: LEADER
W20260929 20:23:13.391312 24489 tablet.cc:2369] T 00000000000000000000000000000000 P cde5a4bc3bab4d4489868351ecc30196: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 20:23:13.393623 24381 raft_consensus.cc:2229] T 00000000000000000000000000000000 P cde5a4bc3bab4d4489868351ecc30196 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 20:23:13.394055 24381 raft_consensus.cc:2258] T 00000000000000000000000000000000 P cde5a4bc3bab4d4489868351ecc30196 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:23:13.394275 24381 tablet_replica.cc:330] T 00000000000000000000000000000000 P cde5a4bc3bab4d4489868351ecc30196: stopping tablet replica
I20260929 20:23:13.400768 24381 master.cc:552] Master@127.17.53.124:36147 shutdown complete.
I20260929 20:23:13.400861 24381 master_runner.cc:305] Clearing existing system tablet
I20260929 20:23:13.401302 24381 fs_manager.cc:359] Metadata directory not provided
I20260929 20:23:13.401412 24381 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:23:13.404350 24381 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.003s
I20260929 20:23:13.405961 24571 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:23:13.406394 24381 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713339161492-17621-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:23:13.406632 24381 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260929 20:23:13.406751 24381 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713339161492-17621-0/minicluster-data/master-2/data,/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713339161492-17621-0/minicluster-data/master-2/wal
uuid: "cde5a4bc3bab4d4489868351ecc30196"
format_stamp: "Formatted at 2026-09-29 20:23:13 on dist-test-slave-zbnc"
server_key: "f956cf17a77d8178139ded474b6cf400"
server_key_iv: "af9d052cfbd6bf249450a40e759e7c3d"
server_key_version: "encryptionkey@0"
I20260929 20:23:13.411841 24381 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P cde5a4bc3bab4d4489868351ecc30196: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:23:13.418951 24381 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P cde5a4bc3bab4d4489868351ecc30196: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260929 20:23:13.419135 24381 log.cc:1196] T 00000000000000000000000000000000 P cde5a4bc3bab4d4489868351ecc30196: Deleting WAL directory at /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713339161492-17621-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260929 20:23:13.419571 24381 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P cde5a4bc3bab4d4489868351ecc30196: Deleting consensus metadata
I20260929 20:23:13.419813 24381 master_runner.cc:315] Copying system tablet from 127.17.53.126:34513
I20260929 20:23:13.428467 24381 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P cde5a4bc3bab4d4489868351ecc30196: tablet copy: Beginning tablet copy session from remote peer at address 127.17.53.126:34513
I20260929 20:23:13.444674 24540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:13.429663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59726 (local address 127.17.53.126:34513)
0929 20:23:13.429891 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0929 20:23:13.429898 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:23:13.430024 (+ 126us) server_negotiation.cc:407] Connection header received
0929 20:23:13.430301 (+ 277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:23:13.430307 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:23:13.430392 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:23:13.430504 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:23:13.431543 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:13.432718 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:13.434708 (+ 1990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:13.435411 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:13.435519 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:13.438118 (+ 2599us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:23:13.438125 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:23:13.438130 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:23:13.438171 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:23:13.441738 (+ 3567us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:23:13.442509 (+ 771us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:23:13.442515 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:23:13.442519 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:23:13.442679 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:23:13.443131 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:23:13.443135 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:23:13.443139 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:23:13.443485 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:23:13.443645 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0929 20:23:13.444087 (+ 442us) server_negotiation.cc:299] Negotiation successful
0929 20:23:13.444308 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108}
I20260929 20:23:13.444687 24577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:13.429348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:34513 (local address 127.0.0.1:59726)
0929 20:23:13.429906 (+ 558us) negotiation.cc:107] Waiting for socket to connect
0929 20:23:13.429931 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0929 20:23:13.430136 (+ 205us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:23:13.430563 (+ 427us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:23:13.430568 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:23:13.431127 (+ 559us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:23:13.431394 (+ 267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:23:13.431406 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:23:13.432878 (+ 1472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:23:13.432884 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:23:13.434537 (+ 1653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:23:13.434553 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:23:13.435801 (+ 1248us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:23:13.435807 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:23:13.436010 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:13.436773 (+ 763us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:23:13.436795 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:23:13.437893 (+ 1098us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:23:13.441896 (+ 4003us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:23:13.441902 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:23:13.441907 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:23:13.442384 (+ 477us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:23:13.442800 (+ 416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:23:13.442805 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:23:13.442809 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:23:13.443010 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:23:13.443620 (+ 610us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:23:13.443625 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:23:13.443818 (+ 193us) client_negotiation.cc:769] Sending connection context
0929 20:23:13.443968 (+ 150us) client_negotiation.cc:240] Negotiation successful
0929 20:23:13.444275 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":410,"thread_start_us":545,"threads_started":1}
I20260929 20:23:13.445716 23837 tablet_copy_service.cc:140] P 94c779fe1aa149a9add4b5b63f72b2df: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer cde5a4bc3bab4d4489868351ecc30196 ({username='oryx', principal='oryx/127.17.53.124@KRBTEST.COM'} at 127.0.0.1:59726)
I20260929 20:23:13.445890 23837 tablet_copy_service.cc:161] P 94c779fe1aa149a9add4b5b63f72b2df: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer cde5a4bc3bab4d4489868351ecc30196 at {username='oryx', principal='oryx/127.17.53.124@KRBTEST.COM'} at 127.0.0.1:59726: session id = cde5a4bc3bab4d4489868351ecc30196-00000000000000000000000000000000
I20260929 20:23:13.450047 23837 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 94c779fe1aa149a9add4b5b63f72b2df: Tablet Copy: opened 0 blocks and 1 log segments
I20260929 20:23:13.452447 24381 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260929 20:23:13.462054 24381 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P cde5a4bc3bab4d4489868351ecc30196: tablet copy: Starting download of 0 data blocks...
I20260929 20:23:13.462579 24381 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P cde5a4bc3bab4d4489868351ecc30196: tablet copy: Starting download of 1 WAL segments...
I20260929 20:23:13.466162 24381 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P cde5a4bc3bab4d4489868351ecc30196: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260929 20:23:13.472913 23837 tablet_copy_service.cc:342] P 94c779fe1aa149a9add4b5b63f72b2df: Request end of tablet copy session cde5a4bc3bab4d4489868351ecc30196-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.17.53.124@KRBTEST.COM'} at 127.0.0.1:59726
I20260929 20:23:13.473237 23837 tablet_copy_service.cc:433] P 94c779fe1aa149a9add4b5b63f72b2df: ending tablet copy session cde5a4bc3bab4d4489868351ecc30196-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer cde5a4bc3bab4d4489868351ecc30196
I20260929 20:23:13.480952 17621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu with pid 23878
I20260929 20:23:13.486527 24381 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:23:13.489475 24578 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:23:13.495996 24579 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:23:13.501039 24381 server_base.cc:1034] running on GCE node
W20260929 20:23:13.501477 24581 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:23:13.501907 24381 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:23:13.502579 24381 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:23:13.503852 24381 hybrid_clock.cc:648] HybridClock initialized: now 1790713393503801 us; error 40 us; skew 500 ppm
I20260929 20:23:13.504135 24381 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:23:13 dist-test-slave-zbnc krb5kdc[23772](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713393, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.53.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:23:13.505599 17621 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
/tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713339161492-17621-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713339161492-17621-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713339161492-17621-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713339161492-17621-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.53.65:46815
--local_ip_for_outbound_sockets=127.17.53.65
--tserver_master_addrs=127.17.53.126:34513
--webserver_port=36325
--webserver_interface=127.17.53.65
--keytab_file=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713339161492-17621-0/krb5kdc/oryx_127.17.53.65.keytab
--principal=oryx/127.17.53.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.53.84:41905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.17.53.126:34513,127.17.53.125:35701
--tserver_master_addrs=127.17.53.126:34513,127.17.53.125:35701,127.17.53.124:36147 with env {KRB5CCNAME=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713339161492-17621-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713339161492-17621-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713339161492-17621-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 20:23:13.506145 24381 init.cc:377] Logged in from keytab as oryx/127.17.53.124@KRBTEST.COM (short username oryx)
I20260929 20:23:13.513746 24381 webserver.cc:460] Webserver started at http://127.17.53.124:40985/ using document root <none> and password file <none>
I20260929 20:23:13.514140 24381 fs_manager.cc:359] Metadata directory not provided
I20260929 20:23:13.514350 24381 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:23:13.517791 24381 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260929 20:23:13.519531 24589 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:23:13.519994 24381 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20260929 20:23:13.520130 24381 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713339161492-17621-0/minicluster-data/master-2/data,/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713339161492-17621-0/minicluster-data/master-2/wal
uuid: "cde5a4bc3bab4d4489868351ecc30196"
format_stamp: "Formatted at 2026-09-29 20:23:13 on dist-test-slave-zbnc"
server_key: "f956cf17a77d8178139ded474b6cf400"
server_key_iv: "af9d052cfbd6bf249450a40e759e7c3d"
server_key_version: "encryptionkey@0"
I20260929 20:23:13.520320 24381 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713339161492-17621-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713339161492-17621-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713339161492-17621-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:23:13.536182 24381 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:23:13.539786 24381 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:23:13.573117 24381 rpc_server.cc:307] RPC server started. Bound to: 127.17.53.124:36147
I20260929 20:23:13.573167 24640 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.53.124:36147 every 8 connection(s)
I20260929 20:23:13.574149 24381 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713339161492-17621-0/minicluster-data/master-2/data/info.pb
I20260929 20:23:13.581466 24641 sys_catalog.cc:263] Verifying existing consensus state
I20260929 20:23:13.587064 24641 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cde5a4bc3bab4d4489868351ecc30196: Bootstrap starting.
I20260929 20:23:13.607837 24641 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cde5a4bc3bab4d4489868351ecc30196: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 20:23:13.608325 24641 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cde5a4bc3bab4d4489868351ecc30196: Bootstrap complete.
I20260929 20:23:13.609205 24641 raft_consensus.cc:348] T 00000000000000000000000000000000 P cde5a4bc3bab4d4489868351ecc30196 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "94c779fe1aa149a9add4b5b63f72b2df" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 34513 } } peers { permanent_uuid: "bbbb7fddd6f64d79bb894cf8c495922e" member_type: VOTER last_known_addr { host: "127.17.53.125" port: 35701 } attrs { promote: false } } peers { permanent_uuid: "cde5a4bc3bab4d4489868351ecc30196" member_type: NON_VOTER last_known_addr { host: "127.17.53.124" port: 36147 } attrs { promote: true } }
I20260929 20:23:13.609470 24641 raft_consensus.cc:729] T 00000000000000000000000000000000 P cde5a4bc3bab4d4489868351ecc30196 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: cde5a4bc3bab4d4489868351ecc30196, State: Initialized, Role: LEARNER
I20260929 20:23:13.609741 24641 consensus_queue.cc:260] T 00000000000000000000000000000000 P cde5a4bc3bab4d4489868351ecc30196 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "94c779fe1aa149a9add4b5b63f72b2df" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 34513 } } peers { permanent_uuid: "bbbb7fddd6f64d79bb894cf8c495922e" member_type: VOTER last_known_addr { host: "127.17.53.125" port: 35701 } attrs { promote: false } } peers { permanent_uuid: "cde5a4bc3bab4d4489868351ecc30196" member_type: NON_VOTER last_known_addr { host: "127.17.53.124" port: 36147 } attrs { promote: true } }
I20260929 20:23:13.611375 24644 sys_catalog.cc:455] T 00000000000000000000000000000000 P cde5a4bc3bab4d4489868351ecc30196 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "94c779fe1aa149a9add4b5b63f72b2df" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 34513 } } peers { permanent_uuid: "bbbb7fddd6f64d79bb894cf8c495922e" member_type: VOTER last_known_addr { host: "127.17.53.125" port: 35701 } attrs { promote: false } } peers { permanent_uuid: "cde5a4bc3bab4d4489868351ecc30196" member_type: NON_VOTER last_known_addr { host: "127.17.53.124" port: 36147 } attrs { promote: true } } }
I20260929 20:23:13.611835 24644 sys_catalog.cc:458] T 00000000000000000000000000000000 P cde5a4bc3bab4d4489868351ecc30196 [sys.catalog]: This master's current role is: LEARNER
I20260929 20:23:13.611706 24641 sys_catalog.cc:564] T 00000000000000000000000000000000 P cde5a4bc3bab4d4489868351ecc30196 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:23:13.625656 24655 catalog_manager.cc:1259] Loaded cluster ID: 8dd81d8e76ec4d3d8e283d6964a8f856
I20260929 20:23:13.625897 24655 catalog_manager.cc:1552] T 00000000000000000000000000000000 P cde5a4bc3bab4d4489868351ecc30196: loading cluster ID for follower catalog manager: success
I20260929 20:23:13.628434 24655 catalog_manager.cc:1574] T 00000000000000000000000000000000 P cde5a4bc3bab4d4489868351ecc30196: acquiring CA information for follower catalog manager: success
I20260929 20:23:13.630244 24655 catalog_manager.cc:1602] T 00000000000000000000000000000000 P cde5a4bc3bab4d4489868351ecc30196: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 20:23:13.782438 24656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:13.759732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44990 (local address 127.17.53.124:36147)
0929 20:23:13.771405 (+ 11673us) server_negotiation.cc:206] Beginning negotiation
0929 20:23:13.771412 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:23:13.771438 (+ 26us) server_negotiation.cc:407] Connection header received
0929 20:23:13.771528 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:23:13.771534 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:23:13.771626 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:23:13.775546 (+ 3920us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:23:13.775822 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:13.777409 (+ 1587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:13.779872 (+ 2463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:13.781283 (+ 1411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:13.781445 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:13.781750 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0929 20:23:13.781904 (+ 154us) server_negotiation.cc:299] Negotiation successful
0929 20:23:13.782058 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11499,"thread_start_us":11292,"threads_started":1}
I20260929 20:23:13.782438 24482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:13.759552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.124:36147 (local address 127.0.0.1:44990)
0929 20:23:13.759821 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0929 20:23:13.759840 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 20:23:13.759964 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:23:13.775106 (+ 15142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:23:13.775112 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:23:13.775171 (+ 59us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:23:13.775364 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:23:13.775378 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:23:13.777602 (+ 2224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:23:13.777609 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:23:13.779661 (+ 2052us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:23:13.779681 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:23:13.781496 (+ 1815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:23:13.781502 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:23:13.781717 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:13.781754 (+ 37us) client_negotiation.cc:769] Sending connection context
0929 20:23:13.781853 (+ 99us) client_negotiation.cc:240] Negotiation successful
0929 20:23:13.782070 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149}
I20260929 20:23:13.783668 24616 raft_consensus.cc:1203] T 00000000000000000000000000000000 P cde5a4bc3bab4d4489868351ecc30196 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260929 20:23:13.784308 24644 sys_catalog.cc:455] T 00000000000000000000000000000000 P cde5a4bc3bab4d4489868351ecc30196 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 94c779fe1aa149a9add4b5b63f72b2df. Latest consensus state: current_term: 1 leader_uuid: "94c779fe1aa149a9add4b5b63f72b2df" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "94c779fe1aa149a9add4b5b63f72b2df" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 34513 } } peers { permanent_uuid: "bbbb7fddd6f64d79bb894cf8c495922e" member_type: VOTER last_known_addr { host: "127.17.53.125" port: 35701 } attrs { promote: false } } peers { permanent_uuid: "cde5a4bc3bab4d4489868351ecc30196" member_type: NON_VOTER last_known_addr { host: "127.17.53.124" port: 36147 } attrs { promote: true } } }
I20260929 20:23:13.784699 24644 sys_catalog.cc:458] T 00000000000000000000000000000000 P cde5a4bc3bab4d4489868351ecc30196 [sys.catalog]: This master's current role is: LEARNER
W20260929 20:23:14.005609 24586 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:23:14.006069 24586 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:23:14.006321 24586 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:23:14.017241 24586 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:23:14.017477 24586 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:23:14.017724 24586 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.53.65
I20260929 20:23:14.031973 24586 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.53.84:41905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713339161492-17621-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713339161492-17621-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713339161492-17621-0/krb5kdc/oryx_127.17.53.65.keytab
--principal=oryx/127.17.53.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.53.65:46815
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713339161492-17621-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.17.53.65
--webserver_port=36325
--tserver_master_addrs=127.17.53.126:34513,127.17.53.125:35701,127.17.53.124:36147
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.53.65
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713339161492-17621-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:23:14.033633 24586 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:23:14.035321 24586 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:23:14.052400 24661 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:23:14.052924 24664 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:23:14.053331 24662 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:23:14.054615 24586 server_base.cc:1034] running on GCE node
I20260929 20:23:14.055589 24586 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:23:14.056851 24586 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:23:14.058118 24586 hybrid_clock.cc:648] HybridClock initialized: now 1790713394058013 us; error 42 us; skew 500 ppm
I20260929 20:23:14.058578 24586 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:23:14 dist-test-slave-zbnc krb5kdc[23772](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713394, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.53.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:23:14.063205 24586 init.cc:377] Logged in from keytab as oryx/127.17.53.65@KRBTEST.COM (short username oryx)
I20260929 20:23:14.065027 24586 webserver.cc:460] Webserver started at http://127.17.53.65:36325/ using document root <none> and password file <none>
I20260929 20:23:14.065802 24586 fs_manager.cc:359] Metadata directory not provided
I20260929 20:23:14.066006 24586 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:23:14.072480 24586 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260929 20:23:14.075870 24671 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:23:14.077364 24586 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 20:23:14.077589 24586 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713339161492-17621-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713339161492-17621-0/minicluster-data/ts-0/wal
uuid: "9bfa2167798b44b1aeb4934b00d01783"
format_stamp: "Formatted at 2026-09-29 20:23:10 on dist-test-slave-zbnc"
server_key: "a7696d9cca47ee27e22753f1870c3495"
server_key_iv: "6d8668926670c223f2f77619efadde53"
server_key_version: "encryptionkey@0"
I20260929 20:23:14.079012 24586 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713339161492-17621-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713339161492-17621-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713339161492-17621-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:23:14.097942 24586 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:23:14.102749 24586 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:23:14.103121 24586 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:23:14.104260 24586 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:23:14.105959 24586 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:23:14.106081 24586 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:23:14.106180 24586 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:23:14.106256 24586 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:23:14.177054 24586 rpc_server.cc:307] RPC server started. Bound to: 127.17.53.65:46815
I20260929 20:23:14.177209 24783 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.53.65:46815 every 8 connection(s)
I20260929 20:23:14.179052 24586 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713339161492-17621-0/minicluster-data/ts-0/data/info.pb
I20260929 20:23:14.185833 24563 sys_catalog.cc:455] T 00000000000000000000000000000000 P 94c779fe1aa149a9add4b5b63f72b2df [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "94c779fe1aa149a9add4b5b63f72b2df" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "94c779fe1aa149a9add4b5b63f72b2df" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 34513 } } peers { permanent_uuid: "bbbb7fddd6f64d79bb894cf8c495922e" member_type: VOTER last_known_addr { host: "127.17.53.125" port: 35701 } attrs { promote: false } } peers { permanent_uuid: "cde5a4bc3bab4d4489868351ecc30196" member_type: NON_VOTER last_known_addr { host: "127.17.53.124" port: 36147 } attrs { promote: true } } }
I20260929 20:23:14.186481 24563 sys_catalog.cc:458] T 00000000000000000000000000000000 P 94c779fe1aa149a9add4b5b63f72b2df [sys.catalog]: This master's current role is: LEADER
I20260929 20:23:14.188458 17621 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu as pid 24586
I20260929 20:23:14.188998 17621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu pid 24012
I20260929 20:23:14.197036 24786 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 94c779fe1aa149a9add4b5b63f72b2df: attempting to promote NON_VOTER cde5a4bc3bab4d4489868351ecc30196 to VOTER
I20260929 20:23:14.198920 24786 consensus_queue.cc:237] T 00000000000000000000000000000000 P 94c779fe1aa149a9add4b5b63f72b2df [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "94c779fe1aa149a9add4b5b63f72b2df" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 34513 } } peers { permanent_uuid: "bbbb7fddd6f64d79bb894cf8c495922e" member_type: VOTER last_known_addr { host: "127.17.53.125" port: 35701 } attrs { promote: false } } peers { permanent_uuid: "cde5a4bc3bab4d4489868351ecc30196" member_type: VOTER last_known_addr { host: "127.17.53.124" port: 36147 } attrs { promote: false } }
I20260929 20:23:14.204423 24616 raft_consensus.cc:1261] T 00000000000000000000000000000000 P cde5a4bc3bab4d4489868351ecc30196 [term 1 LEARNER]: Refusing update from remote peer 94c779fe1aa149a9add4b5b63f72b2df: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260929 20:23:14.205127 24427 raft_consensus.cc:1261] T 00000000000000000000000000000000 P bbbb7fddd6f64d79bb894cf8c495922e [term 1 FOLLOWER]: Refusing update from remote peer 94c779fe1aa149a9add4b5b63f72b2df: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260929 20:23:14.206142 24563 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 94c779fe1aa149a9add4b5b63f72b2df [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbbb7fddd6f64d79bb894cf8c495922e" member_type: VOTER last_known_addr { host: "127.17.53.125" port: 35701 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260929 20:23:14.207561 24563 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 94c779fe1aa149a9add4b5b63f72b2df [LEADER]: Connected to new peer: Peer: permanent_uuid: "cde5a4bc3bab4d4489868351ecc30196" member_type: VOTER last_known_addr { host: "127.17.53.124" port: 36147 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260929 20:23:14.210319 24789 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 94c779fe1aa149a9add4b5b63f72b2df [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, cde5a4bc3bab4d4489868351ecc30196 (127.17.53.124) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "94c779fe1aa149a9add4b5b63f72b2df" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 34513 } } peers { permanent_uuid: "bbbb7fddd6f64d79bb894cf8c495922e" member_type: VOTER last_known_addr { host: "127.17.53.125" port: 35701 } attrs { promote: false } } peers { permanent_uuid: "cde5a4bc3bab4d4489868351ecc30196" member_type: VOTER last_known_addr { host: "127.17.53.124" port: 36147 } attrs { promote: false } } }
I20260929 20:23:14.211140 24427 raft_consensus.cc:2932] T 00000000000000000000000000000000 P bbbb7fddd6f64d79bb894cf8c495922e [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, cde5a4bc3bab4d4489868351ecc30196 (127.17.53.124) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "94c779fe1aa149a9add4b5b63f72b2df" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 34513 } } peers { permanent_uuid: "bbbb7fddd6f64d79bb894cf8c495922e" member_type: VOTER last_known_addr { host: "127.17.53.125" port: 35701 } attrs { promote: false } } peers { permanent_uuid: "cde5a4bc3bab4d4489868351ecc30196" member_type: VOTER last_known_addr { host: "127.17.53.124" port: 36147 } attrs { promote: false } } }
I20260929 20:23:14.228564 24799 sys_catalog.cc:455] T 00000000000000000000000000000000 P bbbb7fddd6f64d79bb894cf8c495922e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "94c779fe1aa149a9add4b5b63f72b2df" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "94c779fe1aa149a9add4b5b63f72b2df" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 34513 } } peers { permanent_uuid: "bbbb7fddd6f64d79bb894cf8c495922e" member_type: VOTER last_known_addr { host: "127.17.53.125" port: 35701 } attrs { promote: false } } peers { permanent_uuid: "cde5a4bc3bab4d4489868351ecc30196" member_type: VOTER last_known_addr { host: "127.17.53.124" port: 36147 } attrs { promote: false } } }
I20260929 20:23:14.229564 24786 sys_catalog.cc:455] T 00000000000000000000000000000000 P 94c779fe1aa149a9add4b5b63f72b2df [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "94c779fe1aa149a9add4b5b63f72b2df" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "94c779fe1aa149a9add4b5b63f72b2df" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 34513 } } peers { permanent_uuid: "bbbb7fddd6f64d79bb894cf8c495922e" member_type: VOTER last_known_addr { host: "127.17.53.125" port: 35701 } attrs { promote: false } } peers { permanent_uuid: "cde5a4bc3bab4d4489868351ecc30196" member_type: VOTER last_known_addr { host: "127.17.53.124" port: 36147 } attrs { promote: false } } }
I20260929 20:23:14.229578 24563 sys_catalog.cc:455] T 00000000000000000000000000000000 P 94c779fe1aa149a9add4b5b63f72b2df [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "94c779fe1aa149a9add4b5b63f72b2df" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "94c779fe1aa149a9add4b5b63f72b2df" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 34513 } } peers { permanent_uuid: "bbbb7fddd6f64d79bb894cf8c495922e" member_type: VOTER last_known_addr { host: "127.17.53.125" port: 35701 } attrs { promote: false } } peers { permanent_uuid: "cde5a4bc3bab4d4489868351ecc30196" member_type: VOTER last_known_addr { host: "127.17.53.124" port: 36147 } attrs { promote: false } } }
I20260929 20:23:14.229967 24786 sys_catalog.cc:458] T 00000000000000000000000000000000 P 94c779fe1aa149a9add4b5b63f72b2df [sys.catalog]: This master's current role is: LEADER
I20260929 20:23:14.230165 24563 sys_catalog.cc:458] T 00000000000000000000000000000000 P 94c779fe1aa149a9add4b5b63f72b2df [sys.catalog]: This master's current role is: LEADER
I20260929 20:23:14.231174 24799 sys_catalog.cc:458] T 00000000000000000000000000000000 P bbbb7fddd6f64d79bb894cf8c495922e [sys.catalog]: This master's current role is: FOLLOWER
I20260929 20:23:14.232890 24563 sys_catalog.cc:455] T 00000000000000000000000000000000 P 94c779fe1aa149a9add4b5b63f72b2df [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "94c779fe1aa149a9add4b5b63f72b2df" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "94c779fe1aa149a9add4b5b63f72b2df" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 34513 } } peers { permanent_uuid: "bbbb7fddd6f64d79bb894cf8c495922e" member_type: VOTER last_known_addr { host: "127.17.53.125" port: 35701 } attrs { promote: false } } peers { permanent_uuid: "cde5a4bc3bab4d4489868351ecc30196" member_type: VOTER last_known_addr { host: "127.17.53.124" port: 36147 } attrs { promote: false } } }
I20260929 20:23:14.233276 24563 sys_catalog.cc:458] T 00000000000000000000000000000000 P 94c779fe1aa149a9add4b5b63f72b2df [sys.catalog]: This master's current role is: LEADER
I20260929 20:23:14.234547 24616 raft_consensus.cc:2932] T 00000000000000000000000000000000 P cde5a4bc3bab4d4489868351ecc30196 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, cde5a4bc3bab4d4489868351ecc30196 (127.17.53.124) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "94c779fe1aa149a9add4b5b63f72b2df" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 34513 } } peers { permanent_uuid: "bbbb7fddd6f64d79bb894cf8c495922e" member_type: VOTER last_known_addr { host: "127.17.53.125" port: 35701 } attrs { promote: false } } peers { permanent_uuid: "cde5a4bc3bab4d4489868351ecc30196" member_type: VOTER last_known_addr { host: "127.17.53.124" port: 36147 } attrs { promote: false } } }
I20260929 20:23:14.240757 24644 sys_catalog.cc:455] T 00000000000000000000000000000000 P cde5a4bc3bab4d4489868351ecc30196 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "94c779fe1aa149a9add4b5b63f72b2df" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "94c779fe1aa149a9add4b5b63f72b2df" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 34513 } } peers { permanent_uuid: "bbbb7fddd6f64d79bb894cf8c495922e" member_type: VOTER last_known_addr { host: "127.17.53.125" port: 35701 } attrs { promote: false } } peers { permanent_uuid: "cde5a4bc3bab4d4489868351ecc30196" member_type: VOTER last_known_addr { host: "127.17.53.124" port: 36147 } attrs { promote: false } } }
I20260929 20:23:14.241106 24644 sys_catalog.cc:458] T 00000000000000000000000000000000 P cde5a4bc3bab4d4489868351ecc30196 [sys.catalog]: This master's current role is: FOLLOWER
Sep 29 20:23:14 dist-test-slave-zbnc krb5kdc[23772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713389, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.17.53.66@KRBTEST.COM
Sep 29 20:23:14 dist-test-slave-zbnc krb5kdc[23772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713394, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.53.65@KRBTEST.COM for oryx/127.17.53.125@KRBTEST.COM
Sep 29 20:23:14 dist-test-slave-zbnc krb5kdc[23772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713394, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.53.65@KRBTEST.COM for oryx/127.17.53.124@KRBTEST.COM
Sep 29 20:23:14 dist-test-slave-zbnc krb5kdc[23772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713394, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.53.65@KRBTEST.COM for oryx/127.17.53.126@KRBTEST.COM
I20260929 20:23:14.260658 24797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:14.201634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.125:35701 (local address 127.17.53.65:48669)
0929 20:23:14.223862 (+ 22228us) negotiation.cc:107] Waiting for socket to connect
0929 20:23:14.223885 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0929 20:23:14.225911 (+ 2026us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:23:14.226659 (+ 748us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:23:14.226667 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:23:14.231416 (+ 4749us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:23:14.231980 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:23:14.232000 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:23:14.234192 (+ 2192us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:23:14.234197 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:23:14.237729 (+ 3532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:23:14.237743 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:23:14.239669 (+ 1926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:23:14.239678 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:23:14.240076 (+ 398us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:14.241089 (+ 1013us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:23:14.241187 (+ 98us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:23:14.248578 (+ 7391us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:23:14.254648 (+ 6070us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:23:14.254670 (+ 22us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:23:14.254698 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:23:14.255697 (+ 999us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:23:14.256085 (+ 388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:23:14.256091 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:23:14.256095 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:23:14.256345 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:23:14.256975 (+ 630us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:23:14.256993 (+ 18us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:23:14.257420 (+ 427us) client_negotiation.cc:769] Sending connection context
0929 20:23:14.257886 (+ 466us) client_negotiation.cc:240] Negotiation successful
0929 20:23:14.258660 (+ 774us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22004,"thread_start_us":22468,"threads_started":1}
I20260929 20:23:14.262387 24796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:14.198798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.53.65:48669 (local address 127.17.53.125:35701)
0929 20:23:14.199721 (+ 923us) server_negotiation.cc:206] Beginning negotiation
0929 20:23:14.199729 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 20:23:14.223994 (+ 24265us) server_negotiation.cc:407] Connection header received
0929 20:23:14.226114 (+ 2120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:23:14.226122 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:23:14.226249 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:23:14.226346 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:23:14.232160 (+ 5814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:14.233875 (+ 1715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:14.237908 (+ 4033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:14.239002 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:14.239162 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:14.249016 (+ 9854us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:23:14.249026 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:23:14.249031 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:23:14.249078 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:23:14.254464 (+ 5386us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:23:14.255806 (+ 1342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:23:14.255812 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:23:14.255815 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:23:14.255971 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:23:14.256466 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:23:14.256471 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:23:14.256474 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:23:14.256804 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:23:14.256965 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0929 20:23:14.261512 (+ 4547us) server_negotiation.cc:299] Negotiation successful
0929 20:23:14.261807 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":760,"thread_start_us":642,"threads_started":1}
I20260929 20:23:14.262404 24656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:14.183193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.53.65:43265 (local address 127.17.53.124:36147)
0929 20:23:14.183392 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0929 20:23:14.183397 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:23:14.221182 (+ 37785us) server_negotiation.cc:407] Connection header received
0929 20:23:14.221996 (+ 814us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:23:14.222005 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:23:14.222130 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:23:14.222260 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:23:14.234317 (+ 12057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:14.235732 (+ 1415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:14.240978 (+ 5246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:14.242340 (+ 1362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:14.242442 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:14.249353 (+ 6911us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:23:14.249361 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:23:14.249366 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:23:14.249417 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:23:14.255313 (+ 5896us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:23:14.258489 (+ 3176us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:23:14.258495 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:23:14.258498 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:23:14.258788 (+ 290us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:23:14.260301 (+ 1513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:23:14.260306 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:23:14.260320 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:23:14.260770 (+ 450us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:23:14.260907 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0929 20:23:14.261524 (+ 617us) server_negotiation.cc:299] Negotiation successful
0929 20:23:14.261840 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260929 20:23:14.263605 24788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:14.184063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.53.65:38615 (local address 127.17.53.126:34513)
0929 20:23:14.184907 (+ 844us) server_negotiation.cc:206] Beginning negotiation
0929 20:23:14.184914 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:23:14.187303 (+ 2389us) server_negotiation.cc:407] Connection header received
0929 20:23:14.224134 (+ 36831us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:23:14.224142 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:23:14.224243 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:23:14.224339 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:23:14.238840 (+ 14501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:14.240436 (+ 1596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:14.245593 (+ 5157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:14.246633 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:14.246743 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:14.252806 (+ 6063us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:23:14.252819 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:23:14.252826 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:23:14.252871 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:23:14.258332 (+ 5461us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:23:14.260200 (+ 1868us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:23:14.260209 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:23:14.260214 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:23:14.260457 (+ 243us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:23:14.260990 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:23:14.260995 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:23:14.260999 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:23:14.261464 (+ 465us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:23:14.261573 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0929 20:23:14.262467 (+ 894us) server_negotiation.cc:299] Negotiation successful
0929 20:23:14.262840 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":583,"thread_start_us":422,"threads_started":1}
I20260929 20:23:14.263672 24794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:14.190906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43260 (local address 127.17.53.66:35805)
0929 20:23:14.221542 (+ 30636us) server_negotiation.cc:206] Beginning negotiation
0929 20:23:14.221556 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 20:23:14.222877 (+ 1321us) server_negotiation.cc:407] Connection header received
0929 20:23:14.222969 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:23:14.222982 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:23:14.223368 (+ 386us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:23:14.223590 (+ 222us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:23:14.231013 (+ 7423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:14.233381 (+ 2368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:14.236408 (+ 3027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:14.237721 (+ 1313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:14.237908 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:14.249684 (+ 11776us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:23:14.249713 (+ 29us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:23:14.249732 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:23:14.249794 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:23:14.257201 (+ 7407us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:23:14.258653 (+ 1452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:23:14.258664 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:23:14.258667 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:23:14.258937 (+ 270us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:23:14.259569 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:23:14.259574 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:23:14.259577 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:23:14.260021 (+ 444us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:23:14.260444 (+ 423us) server_negotiation.cc:1035] Waiting for connection context
0929 20:23:14.262209 (+ 1765us) server_negotiation.cc:299] Negotiation successful
0929 20:23:14.263027 (+ 818us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29305,"thread_start_us":29436,"threads_started":1}
I20260929 20:23:14.264104 24787 heartbeater.cc:347] Connected to a master server at 127.17.53.125:35701
I20260929 20:23:14.264405 24792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:14.184401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.124:36147 (local address 127.17.53.65:43265)
0929 20:23:14.221051 (+ 36650us) negotiation.cc:107] Waiting for socket to connect
0929 20:23:14.221079 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0929 20:23:14.221844 (+ 765us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:23:14.228912 (+ 7068us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:23:14.228918 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:23:14.232926 (+ 4008us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:23:14.234093 (+ 1167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:23:14.234116 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:23:14.236438 (+ 2322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:23:14.236444 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:23:14.238629 (+ 2185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:23:14.238643 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:23:14.243132 (+ 4489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:23:14.243144 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:23:14.243547 (+ 403us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:14.244744 (+ 1197us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:23:14.244768 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:23:14.248994 (+ 4226us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:23:14.257435 (+ 8441us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:23:14.257441 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:23:14.257446 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:23:14.258330 (+ 884us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:23:14.258982 (+ 652us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:23:14.258987 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:23:14.258991 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:23:14.260167 (+ 1176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:23:14.260992 (+ 825us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:23:14.260997 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:23:14.261198 (+ 201us) client_negotiation.cc:769] Sending connection context
0929 20:23:14.262985 (+ 1787us) client_negotiation.cc:240] Negotiation successful
0929 20:23:14.263703 (+ 718us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34594,"thread_start_us":2749,"threads_started":1}
I20260929 20:23:14.264559 24787 heartbeater.cc:464] Registering TS with master...
I20260929 20:23:14.265539 24135 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:23:14.265954 24784 heartbeater.cc:347] Connected to a master server at 127.17.53.124:36147
I20260929 20:23:14.266014 24787 heartbeater.cc:511] Master 127.17.53.125:35701 requested a full tablet report, sending...
I20260929 20:23:14.266278 24784 heartbeater.cc:464] Registering TS with master...
I20260929 20:23:14.267256 24784 heartbeater.cc:511] Master 127.17.53.124:36147 requested a full tablet report, sending...
I20260929 20:23:14.269451 24417 ts_manager.cc:194] Registered new tserver with Master: 9bfa2167798b44b1aeb4934b00d01783 (127.17.53.65:46815)
I20260929 20:23:14.270201 24606 ts_manager.cc:194] Registered new tserver with Master: 9bfa2167798b44b1aeb4934b00d01783 (127.17.53.65:46815)
I20260929 20:23:14.274706 24790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:14.184106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:34513 (local address 127.17.53.65:38615)
0929 20:23:14.187139 (+ 3033us) negotiation.cc:107] Waiting for socket to connect
0929 20:23:14.187199 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0929 20:23:14.218551 (+ 31352us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:23:14.225461 (+ 6910us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:23:14.225477 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:23:14.226746 (+ 1269us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:23:14.233777 (+ 7031us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:23:14.233790 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:23:14.240675 (+ 6885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:23:14.240681 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:23:14.245328 (+ 4647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:23:14.245348 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:23:14.247183 (+ 1835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:23:14.247193 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:23:14.247376 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:14.248131 (+ 755us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:23:14.248155 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:23:14.252178 (+ 4023us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:23:14.258487 (+ 6309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:23:14.258491 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:23:14.258495 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:23:14.260021 (+ 1526us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:23:14.260625 (+ 604us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:23:14.260630 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:23:14.260633 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:23:14.260869 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:23:14.261675 (+ 806us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:23:14.261680 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:23:14.261880 (+ 200us) client_negotiation.cc:769] Sending connection context
0929 20:23:14.273572 (+ 11692us) client_negotiation.cc:240] Negotiation successful
0929 20:23:14.273986 (+ 414us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2516,"spinlock_wait_cycles":1614592,"thread_start_us":34393,"threads_started":1}
I20260929 20:23:14.276007 24785 heartbeater.cc:347] Connected to a master server at 127.17.53.126:34513
I20260929 20:23:14.276201 24785 heartbeater.cc:464] Registering TS with master...
I20260929 20:23:14.277052 24785 heartbeater.cc:511] Master 127.17.53.126:34513 requested a full tablet report, sending...
I20260929 20:23:14.278622 23817 ts_manager.cc:194] Re-registered known tserver with Master: 9bfa2167798b44b1aeb4934b00d01783 (127.17.53.65:46815)
I20260929 20:23:14.280340 23817 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.17.53.65@KRBTEST.COM'} at 127.17.53.65:38615
I20260929 20:23:14.364095 17621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu with pid 24012
I20260929 20:23:14.380671 17621 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
/tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713339161492-17621-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713339161492-17621-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713339161492-17621-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713339161492-17621-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.53.66:35805
--local_ip_for_outbound_sockets=127.17.53.66
--tserver_master_addrs=127.17.53.126:34513
--webserver_port=38109
--webserver_interface=127.17.53.66
--keytab_file=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713339161492-17621-0/krb5kdc/oryx_127.17.53.66.keytab
--principal=oryx/127.17.53.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.53.84:41905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.17.53.126:34513,127.17.53.125:35701
--tserver_master_addrs=127.17.53.126:34513,127.17.53.125:35701,127.17.53.124:36147 with env {KRB5CCNAME=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713339161492-17621-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713339161492-17621-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713339161492-17621-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:23:14.727470 24801 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:23:14.727777 24801 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:23:14.727954 24801 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:23:14.735816 24801 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:23:14.735965 24801 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:23:14.736141 24801 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.53.66
I20260929 20:23:14.746120 24801 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.53.84:41905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713339161492-17621-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713339161492-17621-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713339161492-17621-0/krb5kdc/oryx_127.17.53.66.keytab
--principal=oryx/127.17.53.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.53.66:35805
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713339161492-17621-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.17.53.66
--webserver_port=38109
--tserver_master_addrs=127.17.53.126:34513,127.17.53.125:35701,127.17.53.124:36147
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.53.66
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713339161492-17621-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:23:14.747593 24801 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:23:14.748917 24801 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:23:14.765466 24807 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:23:14.765776 24809 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:23:14.765538 24806 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:23:14.766978 24801 server_base.cc:1034] running on GCE node
I20260929 20:23:14.767588 24801 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:23:14.768692 24801 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:23:14.769966 24801 hybrid_clock.cc:648] HybridClock initialized: now 1790713394769909 us; error 46 us; skew 500 ppm
I20260929 20:23:14.770639 24801 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:23:14 dist-test-slave-zbnc krb5kdc[23772](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713394, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.53.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:23:14.773821 24801 init.cc:377] Logged in from keytab as oryx/127.17.53.66@KRBTEST.COM (short username oryx)
I20260929 20:23:14.775403 24801 webserver.cc:460] Webserver started at http://127.17.53.66:38109/ using document root <none> and password file <none>
I20260929 20:23:14.776135 24801 fs_manager.cc:359] Metadata directory not provided
I20260929 20:23:14.776294 24801 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:23:14.781657 24801 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260929 20:23:14.784672 24816 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:23:14.786136 24801 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 20:23:14.786269 24801 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713339161492-17621-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713339161492-17621-0/minicluster-data/ts-1/wal
uuid: "458a0c68846640e7b375f7500abe9076"
format_stamp: "Formatted at 2026-09-29 20:23:11 on dist-test-slave-zbnc"
server_key: "c15f13e749933846e0f03b63f8379ff2"
server_key_iv: "ae93643beefc362efa1be18970a213a8"
server_key_version: "encryptionkey@0"
I20260929 20:23:14.787282 24801 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713339161492-17621-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713339161492-17621-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713339161492-17621-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:23:14.807734 24801 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:23:14.811920 24801 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:23:14.812209 24801 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:23:14.813308 24801 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:23:14.814674 24801 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:23:14.814759 24801 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:23:14.814834 24801 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:23:14.814924 24801 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260929 20:23:14.847482 24801 rpc_server.cc:307] RPC server started. Bound to: 127.17.53.66:35805
I20260929 20:23:14.847536 24928 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.53.66:35805 every 8 connection(s)
I20260929 20:23:14.848979 24801 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713339161492-17621-0/minicluster-data/ts-1/data/info.pb
I20260929 20:23:14.851862 17621 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu as pid 24801
I20260929 20:23:14.852335 17621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu pid 24146
Sep 29 20:23:14 dist-test-slave-zbnc krb5kdc[23772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713389, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.17.53.67@KRBTEST.COM
Sep 29 20:23:14 dist-test-slave-zbnc krb5kdc[23772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713394, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.53.66@KRBTEST.COM for oryx/127.17.53.124@KRBTEST.COM
Sep 29 20:23:14 dist-test-slave-zbnc krb5kdc[23772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713394, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.53.66@KRBTEST.COM for oryx/127.17.53.126@KRBTEST.COM
I20260929 20:23:14.894825 24938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:14.854690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45016 (local address 127.17.53.67:46647)
0929 20:23:14.863452 (+ 8762us) server_negotiation.cc:206] Beginning negotiation
0929 20:23:14.863466 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 20:23:14.863506 (+ 40us) server_negotiation.cc:407] Connection header received
0929 20:23:14.863605 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:23:14.863618 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:23:14.863951 (+ 333us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:23:14.864143 (+ 192us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:23:14.871703 (+ 7560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:14.872751 (+ 1048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:14.874625 (+ 1874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:14.875519 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:14.875637 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:14.882230 (+ 6593us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:23:14.882247 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:23:14.882269 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:23:14.882320 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:23:14.887570 (+ 5250us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:23:14.888776 (+ 1206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:23:14.888786 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:23:14.888790 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:23:14.889418 (+ 628us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:23:14.890844 (+ 1426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:23:14.890877 (+ 33us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:23:14.890883 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:23:14.892188 (+ 1305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:23:14.892613 (+ 425us) server_negotiation.cc:1035] Waiting for connection context
0929 20:23:14.893124 (+ 511us) server_negotiation.cc:299] Negotiation successful
0929 20:23:14.894086 (+ 962us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8539,"thread_start_us":8351,"threads_started":1}
I20260929 20:23:14.896867 24269 generic_service.cc:196] Checking for leaks (request via RPC)
Sep 29 20:23:14 dist-test-slave-zbnc krb5kdc[23772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713394, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.53.66@KRBTEST.COM for oryx/127.17.53.125@KRBTEST.COM
I20260929 20:23:14.905344 24937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:14.854072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.53.66:53191 (local address 127.17.53.126:34513)
0929 20:23:14.864393 (+ 10321us) server_negotiation.cc:206] Beginning negotiation
0929 20:23:14.864413 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0929 20:23:14.870919 (+ 6506us) server_negotiation.cc:407] Connection header received
0929 20:23:14.879086 (+ 8167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:23:14.879094 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:23:14.879192 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:23:14.879287 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:23:14.882651 (+ 3364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:14.883853 (+ 1202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:14.886776 (+ 2923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:14.887773 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:14.887916 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:14.895792 (+ 7876us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:23:14.895800 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:23:14.895813 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:23:14.895866 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:23:14.901088 (+ 5222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:23:14.902040 (+ 952us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:23:14.902047 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:23:14.902054 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:23:14.902291 (+ 237us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:23:14.902925 (+ 634us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:23:14.902930 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:23:14.902934 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:23:14.903373 (+ 439us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:23:14.903551 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0929 20:23:14.904539 (+ 988us) server_negotiation.cc:299] Negotiation successful
0929 20:23:14.904795 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8611,"thread_start_us":8433,"threads_started":1}
I20260929 20:23:14.907589 24933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:14.851934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.53.66:43015 (local address 127.17.53.124:36147)
0929 20:23:14.861417 (+ 9483us) server_negotiation.cc:206] Beginning negotiation
0929 20:23:14.861427 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 20:23:14.871561 (+ 10134us) server_negotiation.cc:407] Connection header received
0929 20:23:14.873696 (+ 2135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:23:14.873702 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:23:14.873784 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:23:14.873880 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:23:14.876480 (+ 2600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:14.877723 (+ 1243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:14.884036 (+ 6313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:14.885140 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:14.885277 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:14.892147 (+ 6870us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:23:14.892154 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:23:14.892158 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:23:14.892202 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:23:14.898160 (+ 5958us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:23:14.900176 (+ 2016us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:23:14.900183 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:23:14.900187 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:23:14.900531 (+ 344us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:23:14.901427 (+ 896us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:23:14.901432 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:23:14.901435 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:23:14.901910 (+ 475us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:23:14.902545 (+ 635us) server_negotiation.cc:1035] Waiting for connection context
0929 20:23:14.906623 (+ 4078us) server_negotiation.cc:299] Negotiation successful
0929 20:23:14.906884 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9296,"thread_start_us":9477,"threads_started":1}
I20260929 20:23:14.908346 24935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:14.853389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.124:36147 (local address 127.17.53.66:43015)
0929 20:23:14.871424 (+ 18035us) negotiation.cc:107] Waiting for socket to connect
0929 20:23:14.871445 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 20:23:14.873227 (+ 1782us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:23:14.874312 (+ 1085us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:23:14.874325 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:23:14.875350 (+ 1025us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:23:14.876270 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:23:14.876298 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:23:14.877932 (+ 1634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:23:14.877938 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:23:14.883718 (+ 5780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:23:14.883730 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:23:14.885734 (+ 2004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:23:14.885743 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:23:14.886231 (+ 488us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:14.887187 (+ 956us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:23:14.887211 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:23:14.891905 (+ 4694us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:23:14.898445 (+ 6540us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:23:14.898543 (+ 98us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:23:14.898570 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:23:14.899845 (+ 1275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:23:14.900740 (+ 895us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:23:14.900746 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:23:14.900750 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:23:14.901243 (+ 493us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:23:14.902611 (+ 1368us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:23:14.902620 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:23:14.903975 (+ 1355us) client_negotiation.cc:769] Sending connection context
0929 20:23:14.905613 (+ 1638us) client_negotiation.cc:240] Negotiation successful
0929 20:23:14.905932 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17906,"mutex_wait_us":102,"spinlock_wait_cycles":54016,"thread_start_us":17727,"threads_started":1}
I20260929 20:23:14.909169 24934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:14.852697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:34513 (local address 127.17.53.66:53191)
0929 20:23:14.867618 (+ 14921us) negotiation.cc:107] Waiting for socket to connect
0929 20:23:14.867674 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0929 20:23:14.878895 (+ 11221us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:23:14.879518 (+ 623us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:23:14.879524 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:23:14.879997 (+ 473us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:23:14.880323 (+ 326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:23:14.880336 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:23:14.884063 (+ 3727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:23:14.884068 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:23:14.886598 (+ 2530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:23:14.886633 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:23:14.888044 (+ 1411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:23:14.888052 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:23:14.888399 (+ 347us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:14.889414 (+ 1015us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:23:14.889439 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:23:14.895525 (+ 6086us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:23:14.901212 (+ 5687us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:23:14.901218 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:23:14.901222 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:23:14.901926 (+ 704us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:23:14.902421 (+ 495us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:23:14.902429 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:23:14.902435 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:23:14.902795 (+ 360us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:23:14.903507 (+ 712us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:23:14.903543 (+ 36us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:23:14.904027 (+ 484us) client_negotiation.cc:769] Sending connection context
0929 20:23:14.904391 (+ 364us) client_negotiation.cc:240] Negotiation successful
0929 20:23:14.905200 (+ 809us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14575,"mutex_wait_us":32,"spinlock_wait_cycles":87424,"thread_start_us":14330,"threads_started":1}
I20260929 20:23:14.910667 24929 heartbeater.cc:347] Connected to a master server at 127.17.53.124:36147
I20260929 20:23:14.910581 24930 heartbeater.cc:347] Connected to a master server at 127.17.53.126:34513
I20260929 20:23:14.911096 24929 heartbeater.cc:464] Registering TS with master...
I20260929 20:23:14.911096 24930 heartbeater.cc:464] Registering TS with master...
I20260929 20:23:14.912673 24930 heartbeater.cc:511] Master 127.17.53.126:34513 requested a full tablet report, sending...
I20260929 20:23:14.913815 24939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:14.864406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.125:35701 (local address 127.17.53.66:46115)
0929 20:23:14.880905 (+ 16499us) negotiation.cc:107] Waiting for socket to connect
0929 20:23:14.880924 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 20:23:14.881601 (+ 677us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:23:14.882130 (+ 529us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:23:14.882134 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:23:14.884330 (+ 2196us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:23:14.884934 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:23:14.884961 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:23:14.888347 (+ 3386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:23:14.888355 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:23:14.891211 (+ 2856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:23:14.891225 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:23:14.892909 (+ 1684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:23:14.892917 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:23:14.893147 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:14.893868 (+ 721us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:23:14.893890 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:23:14.899885 (+ 5995us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:23:14.906985 (+ 7100us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:23:14.906990 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:23:14.906994 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:23:14.907796 (+ 802us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:23:14.908435 (+ 639us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:23:14.908441 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:23:14.908444 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:23:14.909071 (+ 627us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:23:14.912449 (+ 3378us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:23:14.912455 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:23:14.912657 (+ 202us) client_negotiation.cc:769] Sending connection context
0929 20:23:14.912854 (+ 197us) client_negotiation.cc:240] Negotiation successful
0929 20:23:14.913181 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16362,"mutex_wait_us":37,"thread_start_us":16181,"threads_started":1}
I20260929 20:23:14.914197 24940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:14.865096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.53.66:46115 (local address 127.17.53.125:35701)
0929 20:23:14.881465 (+ 16369us) server_negotiation.cc:206] Beginning negotiation
0929 20:23:14.881472 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:23:14.881494 (+ 22us) server_negotiation.cc:407] Connection header received
0929 20:23:14.881759 (+ 265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:23:14.881766 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:23:14.881846 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:23:14.881944 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:23:14.885194 (+ 3250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:14.888119 (+ 2925us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:14.891403 (+ 3284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:14.892393 (+ 990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:14.892541 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:14.900204 (+ 7663us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:23:14.900212 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:23:14.900216 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:23:14.900290 (+ 74us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:23:14.906826 (+ 6536us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:23:14.908048 (+ 1222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:23:14.908054 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:23:14.908057 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:23:14.908312 (+ 255us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:23:14.910506 (+ 2194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:23:14.910512 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:23:14.910517 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:23:14.911133 (+ 616us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:23:14.911501 (+ 368us) server_negotiation.cc:1035] Waiting for connection context
0929 20:23:14.913035 (+ 1534us) server_negotiation.cc:299] Negotiation successful
0929 20:23:14.913373 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16228,"thread_start_us":16086,"threads_started":1}
I20260929 20:23:14.914595 24929 heartbeater.cc:511] Master 127.17.53.124:36147 requested a full tablet report, sending...
I20260929 20:23:14.914929 23817 ts_manager.cc:194] Re-registered known tserver with Master: 458a0c68846640e7b375f7500abe9076 (127.17.53.66:35805)
I20260929 20:23:14.916229 24606 ts_manager.cc:194] Registered new tserver with Master: 458a0c68846640e7b375f7500abe9076 (127.17.53.66:35805)
I20260929 20:23:14.916419 23817 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.17.53.66@KRBTEST.COM'} at 127.17.53.66:53191
I20260929 20:23:14.916671 24931 heartbeater.cc:347] Connected to a master server at 127.17.53.125:35701
I20260929 20:23:14.916850 24931 heartbeater.cc:464] Registering TS with master...
I20260929 20:23:14.917512 24931 heartbeater.cc:511] Master 127.17.53.125:35701 requested a full tablet report, sending...
I20260929 20:23:14.918828 24417 ts_manager.cc:194] Registered new tserver with Master: 458a0c68846640e7b375f7500abe9076 (127.17.53.66:35805)
I20260929 20:23:14.990883 17621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu with pid 24146
W20260929 20:23:15.007164 23802 connection.cc:535] server connection from 127.17.53.67:56901 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260929 20:23:15.007983 17621 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
/tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713339161492-17621-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713339161492-17621-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713339161492-17621-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713339161492-17621-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.53.67:46647
--local_ip_for_outbound_sockets=127.17.53.67
--tserver_master_addrs=127.17.53.126:34513
--webserver_port=35187
--webserver_interface=127.17.53.67
--keytab_file=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713339161492-17621-0/krb5kdc/oryx_127.17.53.67.keytab
--principal=oryx/127.17.53.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.53.84:41905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.17.53.126:34513,127.17.53.125:35701
--tserver_master_addrs=127.17.53.126:34513,127.17.53.125:35701,127.17.53.124:36147 with env {KRB5CCNAME=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713339161492-17621-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713339161492-17621-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713339161492-17621-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 20:23:15.283548 24785 heartbeater.cc:503] Master 127.17.53.126:34513 was elected leader, sending a full tablet report...
W20260929 20:23:15.350286 24942 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:23:15.350627 24942 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:23:15.350798 24942 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:23:15.358347 24942 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:23:15.358520 24942 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:23:15.358703 24942 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.53.67
I20260929 20:23:15.368738 24942 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.53.84:41905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713339161492-17621-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713339161492-17621-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713339161492-17621-0/krb5kdc/oryx_127.17.53.67.keytab
--principal=oryx/127.17.53.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.53.67:46647
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713339161492-17621-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.17.53.67
--webserver_port=35187
--tserver_master_addrs=127.17.53.126:34513,127.17.53.125:35701,127.17.53.124:36147
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.53.67
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713339161492-17621-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:23:15.370127 24942 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:23:15.371506 24942 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:23:15.385699 24948 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:23:15.385747 24949 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:23:15.386075 24951 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:23:15.386458 24942 server_base.cc:1034] running on GCE node
I20260929 20:23:15.387127 24942 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:23:15.388069 24942 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:23:15.389283 24942 hybrid_clock.cc:648] HybridClock initialized: now 1790713395389238 us; error 41 us; skew 500 ppm
I20260929 20:23:15.389603 24942 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:23:15 dist-test-slave-zbnc krb5kdc[23772](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713395, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.53.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:23:15.393008 24942 init.cc:377] Logged in from keytab as oryx/127.17.53.67@KRBTEST.COM (short username oryx)
I20260929 20:23:15.394538 24942 webserver.cc:460] Webserver started at http://127.17.53.67:35187/ using document root <none> and password file <none>
I20260929 20:23:15.395421 24942 fs_manager.cc:359] Metadata directory not provided
I20260929 20:23:15.395538 24942 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:23:15.401243 24942 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 20:23:15.404062 24958 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:23:15.405483 24942 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 20:23:15.405705 24942 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713339161492-17621-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713339161492-17621-0/minicluster-data/ts-2/wal
uuid: "cd595f79664a402192976b9a664b71e7"
format_stamp: "Formatted at 2026-09-29 20:23:11 on dist-test-slave-zbnc"
server_key: "6f7e145f6b441674d81b4fa8bf810520"
server_key_iv: "48ba61e516b2be24afd24995eaa8110f"
server_key_version: "encryptionkey@0"
I20260929 20:23:15.406698 24942 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713339161492-17621-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713339161492-17621-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713339161492-17621-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:23:15.424597 24942 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:23:15.428922 24942 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:23:15.429204 24942 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:23:15.430171 24942 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:23:15.431594 24942 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:23:15.431728 24942 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:23:15.431852 24942 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:23:15.431947 24942 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:23:15.464804 24942 rpc_server.cc:307] RPC server started. Bound to: 127.17.53.67:46647
I20260929 20:23:15.464890 25070 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.53.67:46647 every 8 connection(s)
I20260929 20:23:15.466293 24942 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713339161492-17621-0/minicluster-data/ts-2/data/info.pb
I20260929 20:23:15.471043 17621 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu as pid 24942
Sep 29 20:23:15 dist-test-slave-zbnc krb5kdc[23772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713395, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.53.67@KRBTEST.COM for oryx/127.17.53.125@KRBTEST.COM
Sep 29 20:23:15 dist-test-slave-zbnc krb5kdc[23772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713395, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.53.67@KRBTEST.COM for oryx/127.17.53.126@KRBTEST.COM
Sep 29 20:23:15 dist-test-slave-zbnc krb5kdc[23772](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713395, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.53.67@KRBTEST.COM for oryx/127.17.53.124@KRBTEST.COM
I20260929 20:23:15.514571 25080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:15.470601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.125:35701 (local address 127.17.53.67:60801)
0929 20:23:15.485947 (+ 15346us) negotiation.cc:107] Waiting for socket to connect
0929 20:23:15.485971 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0929 20:23:15.486120 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:23:15.488711 (+ 2591us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:23:15.488719 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:23:15.490125 (+ 1406us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:23:15.490537 (+ 412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:23:15.490567 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:23:15.491855 (+ 1288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:23:15.491862 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:23:15.494500 (+ 2638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:23:15.494514 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:23:15.495861 (+ 1347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:23:15.495868 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:23:15.496156 (+ 288us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:15.496957 (+ 801us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:23:15.496985 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:23:15.500991 (+ 4006us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:23:15.507010 (+ 6019us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:23:15.507023 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:23:15.507047 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:23:15.508093 (+ 1046us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:23:15.509601 (+ 1508us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:23:15.509607 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:23:15.509610 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:23:15.509880 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:23:15.510699 (+ 819us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:23:15.510711 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:23:15.511668 (+ 957us) client_negotiation.cc:769] Sending connection context
0929 20:23:15.512087 (+ 419us) client_negotiation.cc:240] Negotiation successful
0929 20:23:15.512739 (+ 652us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15172,"mutex_wait_us":224,"thread_start_us":18716,"threads_started":1}
I20260929 20:23:15.516314 25075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:15.469211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.53.67:60801 (local address 127.17.53.125:35701)
0929 20:23:15.488301 (+ 19090us) server_negotiation.cc:206] Beginning negotiation
0929 20:23:15.488308 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:23:15.488330 (+ 22us) server_negotiation.cc:407] Connection header received
0929 20:23:15.488389 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:23:15.488393 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:23:15.488475 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:23:15.489033 (+ 558us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:23:15.490800 (+ 1767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:15.491705 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:15.494629 (+ 2924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:15.495248 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:15.495365 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:15.501337 (+ 5972us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:23:15.501344 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:23:15.501359 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:23:15.501400 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:23:15.506790 (+ 5390us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:23:15.508313 (+ 1523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:23:15.508322 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:23:15.508328 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:23:15.508623 (+ 295us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:23:15.510008 (+ 1385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:23:15.510014 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:23:15.510020 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:23:15.510432 (+ 412us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:23:15.510591 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0929 20:23:15.515449 (+ 4858us) server_negotiation.cc:299] Negotiation successful
0929 20:23:15.515747 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18953,"thread_start_us":18812,"threads_started":1}
I20260929 20:23:15.517956 25073 heartbeater.cc:347] Connected to a master server at 127.17.53.125:35701
I20260929 20:23:15.518361 25073 heartbeater.cc:464] Registering TS with master...
I20260929 20:23:15.519652 25073 heartbeater.cc:511] Master 127.17.53.125:35701 requested a full tablet report, sending...
I20260929 20:23:15.522294 24417 ts_manager.cc:194] Registered new tserver with Master: cd595f79664a402192976b9a664b71e7 (127.17.53.67:46647)
I20260929 20:23:15.526940 25077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:15.469554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.53.67:58839 (local address 127.17.53.124:36147)
0929 20:23:15.486884 (+ 17330us) server_negotiation.cc:206] Beginning negotiation
0929 20:23:15.486894 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 20:23:15.489813 (+ 2919us) server_negotiation.cc:407] Connection header received
0929 20:23:15.490074 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:23:15.490082 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:23:15.490158 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:23:15.490242 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:23:15.492982 (+ 2740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:15.494075 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:15.496870 (+ 2795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:15.497527 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:15.497706 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:15.513159 (+ 15453us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:23:15.513170 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:23:15.513178 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:23:15.513246 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:23:15.519759 (+ 6513us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:23:15.523863 (+ 4104us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:23:15.523883 (+ 20us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:23:15.523887 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:23:15.524159 (+ 272us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:23:15.524849 (+ 690us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:23:15.524854 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:23:15.524857 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:23:15.525442 (+ 585us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:23:15.525682 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0929 20:23:15.526318 (+ 636us) server_negotiation.cc:299] Negotiation successful
0929 20:23:15.526515 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17162,"thread_start_us":17053,"threads_started":1}
I20260929 20:23:15.527259 25079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:15.469918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.124:36147 (local address 127.17.53.67:58839)
0929 20:23:15.489706 (+ 19788us) negotiation.cc:107] Waiting for socket to connect
0929 20:23:15.489725 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 20:23:15.489923 (+ 198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:23:15.490512 (+ 589us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:23:15.490518 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:23:15.492428 (+ 1910us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:23:15.492808 (+ 380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:23:15.492820 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:23:15.494257 (+ 1437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:23:15.494263 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:23:15.496666 (+ 2403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:23:15.496677 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:23:15.497987 (+ 1310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:23:15.497993 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:23:15.498284 (+ 291us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:15.499949 (+ 1665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:23:15.499981 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:23:15.512779 (+ 12798us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:23:15.521103 (+ 8324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:23:15.521108 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:23:15.521111 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:23:15.522184 (+ 1073us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:23:15.524314 (+ 2130us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:23:15.524322 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:23:15.524326 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:23:15.524658 (+ 332us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:23:15.525778 (+ 1120us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:23:15.525792 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:23:15.526012 (+ 220us) client_negotiation.cc:769] Sending connection context
0929 20:23:15.526212 (+ 200us) client_negotiation.cc:240] Negotiation successful
0929 20:23:15.526580 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19623,"mutex_wait_us":1531,"thread_start_us":14708,"threads_started":1}
I20260929 20:23:15.528242 25076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:15.469220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.53.67:42393 (local address 127.17.53.126:34513)
0929 20:23:15.487336 (+ 18116us) server_negotiation.cc:206] Beginning negotiation
0929 20:23:15.487345 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 20:23:15.487371 (+ 26us) server_negotiation.cc:407] Connection header received
0929 20:23:15.487452 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:23:15.487460 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:23:15.487588 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:23:15.487736 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:23:15.490972 (+ 3236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:15.492069 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:15.494781 (+ 2712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:15.495462 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:15.495588 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:15.511048 (+ 15460us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:23:15.511065 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:23:15.511070 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:23:15.511148 (+ 78us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:23:15.517155 (+ 6007us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:23:15.521405 (+ 4250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:23:15.521411 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:23:15.521414 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:23:15.521636 (+ 222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:23:15.522935 (+ 1299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:23:15.522940 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:23:15.522943 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:23:15.523519 (+ 576us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:23:15.523646 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0929 20:23:15.527552 (+ 3906us) server_negotiation.cc:299] Negotiation successful
0929 20:23:15.527827 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17952,"thread_start_us":17830,"threads_started":1}
I20260929 20:23:15.528532 25071 heartbeater.cc:347] Connected to a master server at 127.17.53.124:36147
I20260929 20:23:15.528767 25071 heartbeater.cc:464] Registering TS with master...
I20260929 20:23:15.529765 25078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:15.469917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:34513 (local address 127.17.53.67:42393)
0929 20:23:15.476331 (+ 6414us) negotiation.cc:107] Waiting for socket to connect
0929 20:23:15.476401 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0929 20:23:15.479602 (+ 3201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:23:15.488020 (+ 8418us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:23:15.488039 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:23:15.489100 (+ 1061us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:23:15.490756 (+ 1656us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:23:15.490771 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:23:15.492251 (+ 1480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:23:15.492258 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:23:15.494547 (+ 2289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:23:15.494566 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:23:15.499119 (+ 4553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:23:15.499129 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:23:15.499367 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:15.500929 (+ 1562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:23:15.500949 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:23:15.510577 (+ 9628us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:23:15.519963 (+ 9386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:23:15.519969 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:23:15.519972 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:23:15.520926 (+ 954us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:23:15.522346 (+ 1420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:23:15.522352 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:23:15.522357 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:23:15.522731 (+ 374us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:23:15.523647 (+ 916us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:23:15.523652 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:23:15.526921 (+ 3269us) client_negotiation.cc:769] Sending connection context
0929 20:23:15.528500 (+ 1579us) client_negotiation.cc:240] Negotiation successful
0929 20:23:15.528950 (+ 450us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":669,"mutex_wait_us":29,"thread_start_us":578,"threads_started":1}
I20260929 20:23:15.529935 25071 heartbeater.cc:511] Master 127.17.53.124:36147 requested a full tablet report, sending...
I20260929 20:23:15.530901 25072 heartbeater.cc:347] Connected to a master server at 127.17.53.126:34513
I20260929 20:23:15.531087 25072 heartbeater.cc:464] Registering TS with master...
I20260929 20:23:15.531723 24606 ts_manager.cc:194] Registered new tserver with Master: cd595f79664a402192976b9a664b71e7 (127.17.53.67:46647)
I20260929 20:23:15.531932 25072 heartbeater.cc:511] Master 127.17.53.126:34513 requested a full tablet report, sending...
I20260929 20:23:15.533233 23817 ts_manager.cc:194] Re-registered known tserver with Master: cd595f79664a402192976b9a664b71e7 (127.17.53.67:46647)
I20260929 20:23:15.534544 23817 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.17.53.67@KRBTEST.COM'} at 127.17.53.67:42393
I20260929 20:23:15.547705 25086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:15.493286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59728 (local address 127.17.53.126:34513)
0929 20:23:15.511271 (+ 17985us) server_negotiation.cc:206] Beginning negotiation
0929 20:23:15.511283 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 20:23:15.511312 (+ 29us) server_negotiation.cc:407] Connection header received
0929 20:23:15.511399 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:23:15.511405 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:23:15.511486 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:23:15.513314 (+ 1828us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:23:15.523828 (+ 10514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:15.525770 (+ 1942us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:15.528773 (+ 3003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:15.530225 (+ 1452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:15.530352 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:15.536836 (+ 6484us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:23:15.536843 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:23:15.536847 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:23:15.536885 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:23:15.542656 (+ 5771us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:23:15.544029 (+ 1373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:23:15.544037 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:23:15.544041 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:23:15.544333 (+ 292us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:23:15.545247 (+ 914us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:23:15.545254 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:23:15.545260 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:23:15.545746 (+ 486us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:23:15.546102 (+ 356us) server_negotiation.cc:1035] Waiting for connection context
0929 20:23:15.546656 (+ 554us) server_negotiation.cc:299] Negotiation successful
0929 20:23:15.547008 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17753,"thread_start_us":17501,"threads_started":1}
I20260929 20:23:15.552130 25088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:15.496079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45004 (local address 127.17.53.124:36147)
0929 20:23:15.509428 (+ 13349us) server_negotiation.cc:206] Beginning negotiation
0929 20:23:15.509442 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 20:23:15.517336 (+ 7894us) server_negotiation.cc:407] Connection header received
0929 20:23:15.517412 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:23:15.517427 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:23:15.517543 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:23:15.517694 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:23:15.531367 (+ 13673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:15.533633 (+ 2266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:15.536093 (+ 2460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:15.536728 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:15.536860 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:15.542246 (+ 5386us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:23:15.542255 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:23:15.542261 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:23:15.542315 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:23:15.547232 (+ 4917us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:23:15.548471 (+ 1239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:23:15.548477 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:23:15.548480 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:23:15.548742 (+ 262us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:23:15.549486 (+ 744us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:23:15.549491 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:23:15.549495 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:23:15.549955 (+ 460us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:23:15.550309 (+ 354us) server_negotiation.cc:1035] Waiting for connection context
0929 20:23:15.551233 (+ 924us) server_negotiation.cc:299] Negotiation successful
0929 20:23:15.551549 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13157,"thread_start_us":12970,"threads_started":1}
I20260929 20:23:15.554492 25089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:15.503551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48072 (local address 127.17.53.125:35701)
0929 20:23:15.509548 (+ 5997us) server_negotiation.cc:206] Beginning negotiation
0929 20:23:15.509554 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:23:15.521121 (+ 11567us) server_negotiation.cc:407] Connection header received
0929 20:23:15.521485 (+ 364us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:23:15.521491 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:23:15.521607 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:23:15.521722 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:23:15.536811 (+ 15089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:15.538526 (+ 1715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:15.540375 (+ 1849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:15.541062 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:15.541141 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:15.545573 (+ 4432us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:23:15.545578 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:23:15.545581 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:23:15.545616 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:23:15.550167 (+ 4551us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:23:15.551391 (+ 1224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:23:15.551396 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:23:15.551413 (+ 17us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:23:15.551653 (+ 240us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:23:15.552172 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:23:15.552177 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:23:15.552180 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:23:15.552897 (+ 717us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:23:15.553208 (+ 311us) server_negotiation.cc:1035] Waiting for connection context
0929 20:23:15.553847 (+ 639us) server_negotiation.cc:299] Negotiation successful
0929 20:23:15.554036 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5847,"thread_start_us":6475,"threads_started":1}
I20260929 20:23:15.553491 23817 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59728:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 20:23:15.564535 23817 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 20:23:15.616356 25105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:15.600690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45028 (local address 127.17.53.67:46647)
0929 20:23:15.603607 (+ 2917us) server_negotiation.cc:206] Beginning negotiation
0929 20:23:15.603614 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:23:15.603652 (+ 38us) server_negotiation.cc:407] Connection header received
0929 20:23:15.603765 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:23:15.603774 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:23:15.604110 (+ 336us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:23:15.605466 (+ 1356us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:23:15.606126 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:15.608314 (+ 2188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:15.612438 (+ 4124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:15.614847 (+ 2409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:15.615049 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:15.615332 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0929 20:23:15.615708 (+ 376us) server_negotiation.cc:299] Negotiation successful
0929 20:23:15.615912 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2738,"thread_start_us":2467,"threads_started":1}
I20260929 20:23:15.616433 25104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:15.599974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.67:46647 (local address 127.0.0.1:45028)
0929 20:23:15.601249 (+ 1275us) negotiation.cc:107] Waiting for socket to connect
0929 20:23:15.601286 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0929 20:23:15.601438 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:23:15.604359 (+ 2921us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:23:15.604364 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:23:15.604388 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:23:15.605174 (+ 786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:23:15.605207 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:23:15.608480 (+ 3273us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:23:15.608486 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:23:15.612178 (+ 3692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:23:15.612197 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:23:15.615154 (+ 2957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:23:15.615173 (+ 19us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:23:15.615420 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:15.615486 (+ 66us) client_negotiation.cc:769] Sending connection context
0929 20:23:15.615641 (+ 155us) client_negotiation.cc:240] Negotiation successful
0929 20:23:15.615825 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1052,"mutex_wait_us":56,"thread_start_us":799,"threads_started":1}
I20260929 20:23:15.618144 25102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:15.598024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.66:35805 (local address 127.0.0.1:43262)
0929 20:23:15.600894 (+ 2870us) negotiation.cc:107] Waiting for socket to connect
0929 20:23:15.600920 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0929 20:23:15.603613 (+ 2693us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:23:15.604691 (+ 1078us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:23:15.604695 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:23:15.604719 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:23:15.605979 (+ 1260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:23:15.606017 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:23:15.609507 (+ 3490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:23:15.609544 (+ 37us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:23:15.614245 (+ 4701us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:23:15.614271 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:23:15.617006 (+ 2735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:23:15.617013 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:23:15.617291 (+ 278us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:15.617322 (+ 31us) client_negotiation.cc:769] Sending connection context
0929 20:23:15.617404 (+ 82us) client_negotiation.cc:240] Negotiation successful
0929 20:23:15.617532 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2632,"mutex_wait_us":89,"thread_start_us":3197,"threads_started":1}
I20260929 20:23:15.618242 25106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:15.601272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.65:46815 (local address 127.0.0.1:45890)
0929 20:23:15.602485 (+ 1213us) negotiation.cc:107] Waiting for socket to connect
0929 20:23:15.602502 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 20:23:15.602638 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:23:15.605210 (+ 2572us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:23:15.605215 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:23:15.605251 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:23:15.606422 (+ 1171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:23:15.606434 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:23:15.609188 (+ 2754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:23:15.609195 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:23:15.613869 (+ 4674us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:23:15.613895 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:23:15.617239 (+ 3344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:23:15.617245 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:23:15.617399 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:15.617425 (+ 26us) client_negotiation.cc:769] Sending connection context
0929 20:23:15.617584 (+ 159us) client_negotiation.cc:240] Negotiation successful
0929 20:23:15.617697 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1061,"mutex_wait_us":125,"thread_start_us":1750,"threads_started":1}
I20260929 20:23:15.618847 25103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:15.598640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43262 (local address 127.17.53.66:35805)
0929 20:23:15.599545 (+ 905us) server_negotiation.cc:206] Beginning negotiation
0929 20:23:15.599555 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 20:23:15.601639 (+ 2084us) server_negotiation.cc:407] Connection header received
0929 20:23:15.603844 (+ 2205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:23:15.603851 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:23:15.604110 (+ 259us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:23:15.604351 (+ 241us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:23:15.607218 (+ 2867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:15.609322 (+ 2104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:15.614471 (+ 5149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:15.616256 (+ 1785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:15.616406 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:15.616733 (+ 327us) server_negotiation.cc:1035] Waiting for connection context
0929 20:23:15.618420 (+ 1687us) server_negotiation.cc:299] Negotiation successful
0929 20:23:15.618619 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":701,"thread_start_us":568,"threads_started":1}
I20260929 20:23:15.619560 25107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:15.601621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45890 (local address 127.17.53.65:46815)
0929 20:23:15.604593 (+ 2972us) server_negotiation.cc:206] Beginning negotiation
0929 20:23:15.604600 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:23:15.604626 (+ 26us) server_negotiation.cc:407] Connection header received
0929 20:23:15.604718 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:23:15.604724 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:23:15.604937 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:23:15.605231 (+ 294us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:23:15.606737 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:15.609028 (+ 2291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:15.614125 (+ 5097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:15.616468 (+ 2343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:15.616663 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:15.616994 (+ 331us) server_negotiation.cc:1035] Waiting for connection context
0929 20:23:15.618549 (+ 1555us) server_negotiation.cc:299] Negotiation successful
0929 20:23:15.618786 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2762,"thread_start_us":2551,"threads_started":1}
I20260929 20:23:15.622712 24864 tablet_service.cc:1467] Processing CreateTablet for tablet 0e00ee99c5f7428aa7f5976cc5925293 (DEFAULT_TABLE table=test-table [id=67639125246d4b2ba5efff4041cd0f45]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:23:15.622330 25006 tablet_service.cc:1467] Processing CreateTablet for tablet 0e00ee99c5f7428aa7f5976cc5925293 (DEFAULT_TABLE table=test-table [id=67639125246d4b2ba5efff4041cd0f45]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:23:15.624284 24864 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e00ee99c5f7428aa7f5976cc5925293. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:23:15.624522 25006 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e00ee99c5f7428aa7f5976cc5925293. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:23:15.624215 24719 tablet_service.cc:1467] Processing CreateTablet for tablet 0e00ee99c5f7428aa7f5976cc5925293 (DEFAULT_TABLE table=test-table [id=67639125246d4b2ba5efff4041cd0f45]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:23:15.625669 24719 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e00ee99c5f7428aa7f5976cc5925293. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:23:15.637790 25108 tablet_bootstrap.cc:492] T 0e00ee99c5f7428aa7f5976cc5925293 P 458a0c68846640e7b375f7500abe9076: Bootstrap starting.
I20260929 20:23:15.642128 25109 tablet_bootstrap.cc:492] T 0e00ee99c5f7428aa7f5976cc5925293 P cd595f79664a402192976b9a664b71e7: Bootstrap starting.
I20260929 20:23:15.643355 25108 tablet_bootstrap.cc:654] T 0e00ee99c5f7428aa7f5976cc5925293 P 458a0c68846640e7b375f7500abe9076: Neither blocks nor log segments found. Creating new log.
I20260929 20:23:15.645021 25108 log.cc:824] T 0e00ee99c5f7428aa7f5976cc5925293 P 458a0c68846640e7b375f7500abe9076: Log is configured to *not* fsync() on all Append() calls
I20260929 20:23:15.646601 25109 tablet_bootstrap.cc:654] T 0e00ee99c5f7428aa7f5976cc5925293 P cd595f79664a402192976b9a664b71e7: Neither blocks nor log segments found. Creating new log.
I20260929 20:23:15.646770 25110 tablet_bootstrap.cc:492] T 0e00ee99c5f7428aa7f5976cc5925293 P 9bfa2167798b44b1aeb4934b00d01783: Bootstrap starting.
I20260929 20:23:15.648190 25109 log.cc:824] T 0e00ee99c5f7428aa7f5976cc5925293 P cd595f79664a402192976b9a664b71e7: Log is configured to *not* fsync() on all Append() calls
I20260929 20:23:15.648734 25108 tablet_bootstrap.cc:492] T 0e00ee99c5f7428aa7f5976cc5925293 P 458a0c68846640e7b375f7500abe9076: No bootstrap required, opened a new log
I20260929 20:23:15.649168 25108 ts_tablet_manager.cc:1397] T 0e00ee99c5f7428aa7f5976cc5925293 P 458a0c68846640e7b375f7500abe9076: Time spent bootstrapping tablet: real 0.012s user 0.009s sys 0.000s
I20260929 20:23:15.651728 25110 tablet_bootstrap.cc:654] T 0e00ee99c5f7428aa7f5976cc5925293 P 9bfa2167798b44b1aeb4934b00d01783: Neither blocks nor log segments found. Creating new log.
I20260929 20:23:15.652135 25109 tablet_bootstrap.cc:492] T 0e00ee99c5f7428aa7f5976cc5925293 P cd595f79664a402192976b9a664b71e7: No bootstrap required, opened a new log
I20260929 20:23:15.652438 25109 ts_tablet_manager.cc:1397] T 0e00ee99c5f7428aa7f5976cc5925293 P cd595f79664a402192976b9a664b71e7: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.000s
I20260929 20:23:15.653374 25110 log.cc:824] T 0e00ee99c5f7428aa7f5976cc5925293 P 9bfa2167798b44b1aeb4934b00d01783: Log is configured to *not* fsync() on all Append() calls
I20260929 20:23:15.656790 25110 tablet_bootstrap.cc:492] T 0e00ee99c5f7428aa7f5976cc5925293 P 9bfa2167798b44b1aeb4934b00d01783: No bootstrap required, opened a new log
I20260929 20:23:15.657063 25110 ts_tablet_manager.cc:1397] T 0e00ee99c5f7428aa7f5976cc5925293 P 9bfa2167798b44b1aeb4934b00d01783: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.004s
I20260929 20:23:15.658141 25108 raft_consensus.cc:348] T 0e00ee99c5f7428aa7f5976cc5925293 P 458a0c68846640e7b375f7500abe9076 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "458a0c68846640e7b375f7500abe9076" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 35805 } } peers { permanent_uuid: "cd595f79664a402192976b9a664b71e7" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 46647 } } peers { permanent_uuid: "9bfa2167798b44b1aeb4934b00d01783" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 46815 } }
I20260929 20:23:15.658488 25108 raft_consensus.cc:374] T 0e00ee99c5f7428aa7f5976cc5925293 P 458a0c68846640e7b375f7500abe9076 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:23:15.658618 25108 raft_consensus.cc:729] T 0e00ee99c5f7428aa7f5976cc5925293 P 458a0c68846640e7b375f7500abe9076 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 458a0c68846640e7b375f7500abe9076, State: Initialized, Role: FOLLOWER
I20260929 20:23:15.659219 25108 consensus_queue.cc:260] T 0e00ee99c5f7428aa7f5976cc5925293 P 458a0c68846640e7b375f7500abe9076 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "458a0c68846640e7b375f7500abe9076" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 35805 } } peers { permanent_uuid: "cd595f79664a402192976b9a664b71e7" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 46647 } } peers { permanent_uuid: "9bfa2167798b44b1aeb4934b00d01783" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 46815 } }
I20260929 20:23:15.660526 24930 heartbeater.cc:503] Master 127.17.53.126:34513 was elected leader, sending a full tablet report...
I20260929 20:23:15.661100 25108 ts_tablet_manager.cc:1428] T 0e00ee99c5f7428aa7f5976cc5925293 P 458a0c68846640e7b375f7500abe9076: Time spent starting tablet: real 0.012s user 0.006s sys 0.005s
I20260929 20:23:15.666436 25109 raft_consensus.cc:348] T 0e00ee99c5f7428aa7f5976cc5925293 P cd595f79664a402192976b9a664b71e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "458a0c68846640e7b375f7500abe9076" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 35805 } } peers { permanent_uuid: "cd595f79664a402192976b9a664b71e7" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 46647 } } peers { permanent_uuid: "9bfa2167798b44b1aeb4934b00d01783" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 46815 } }
I20260929 20:23:15.666913 25109 raft_consensus.cc:374] T 0e00ee99c5f7428aa7f5976cc5925293 P cd595f79664a402192976b9a664b71e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:23:15.667016 25109 raft_consensus.cc:729] T 0e00ee99c5f7428aa7f5976cc5925293 P cd595f79664a402192976b9a664b71e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd595f79664a402192976b9a664b71e7, State: Initialized, Role: FOLLOWER
I20260929 20:23:15.667616 25109 consensus_queue.cc:260] T 0e00ee99c5f7428aa7f5976cc5925293 P cd595f79664a402192976b9a664b71e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "458a0c68846640e7b375f7500abe9076" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 35805 } } peers { permanent_uuid: "cd595f79664a402192976b9a664b71e7" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 46647 } } peers { permanent_uuid: "9bfa2167798b44b1aeb4934b00d01783" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 46815 } }
I20260929 20:23:15.670768 25110 raft_consensus.cc:348] T 0e00ee99c5f7428aa7f5976cc5925293 P 9bfa2167798b44b1aeb4934b00d01783 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "458a0c68846640e7b375f7500abe9076" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 35805 } } peers { permanent_uuid: "cd595f79664a402192976b9a664b71e7" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 46647 } } peers { permanent_uuid: "9bfa2167798b44b1aeb4934b00d01783" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 46815 } }
I20260929 20:23:15.671268 25110 raft_consensus.cc:374] T 0e00ee99c5f7428aa7f5976cc5925293 P 9bfa2167798b44b1aeb4934b00d01783 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:23:15.671542 25072 heartbeater.cc:503] Master 127.17.53.126:34513 was elected leader, sending a full tablet report...
I20260929 20:23:15.672339 25109 ts_tablet_manager.cc:1428] T 0e00ee99c5f7428aa7f5976cc5925293 P cd595f79664a402192976b9a664b71e7: Time spent starting tablet: real 0.020s user 0.014s sys 0.004s
I20260929 20:23:15.672883 25110 raft_consensus.cc:729] T 0e00ee99c5f7428aa7f5976cc5925293 P 9bfa2167798b44b1aeb4934b00d01783 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bfa2167798b44b1aeb4934b00d01783, State: Initialized, Role: FOLLOWER
I20260929 20:23:15.673564 25110 consensus_queue.cc:260] T 0e00ee99c5f7428aa7f5976cc5925293 P 9bfa2167798b44b1aeb4934b00d01783 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "458a0c68846640e7b375f7500abe9076" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 35805 } } peers { permanent_uuid: "cd595f79664a402192976b9a664b71e7" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 46647 } } peers { permanent_uuid: "9bfa2167798b44b1aeb4934b00d01783" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 46815 } }
I20260929 20:23:15.677552 25110 ts_tablet_manager.cc:1428] T 0e00ee99c5f7428aa7f5976cc5925293 P 9bfa2167798b44b1aeb4934b00d01783: Time spent starting tablet: real 0.020s user 0.016s sys 0.000s
W20260929 20:23:15.719192 25074 tablet.cc:2369] T 0e00ee99c5f7428aa7f5976cc5925293 P cd595f79664a402192976b9a664b71e7: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 20:23:15.724197 24795 tablet.cc:2369] T 0e00ee99c5f7428aa7f5976cc5925293 P 9bfa2167798b44b1aeb4934b00d01783: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 20:23:15.787029 25114 raft_consensus.cc:482] T 0e00ee99c5f7428aa7f5976cc5925293 P 458a0c68846640e7b375f7500abe9076 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 20:23:15.787350 25114 raft_consensus.cc:504] T 0e00ee99c5f7428aa7f5976cc5925293 P 458a0c68846640e7b375f7500abe9076 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "458a0c68846640e7b375f7500abe9076" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 35805 } } peers { permanent_uuid: "cd595f79664a402192976b9a664b71e7" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 46647 } } peers { permanent_uuid: "9bfa2167798b44b1aeb4934b00d01783" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 46815 } }
I20260929 20:23:15.789113 25114 leader_election.cc:288] T 0e00ee99c5f7428aa7f5976cc5925293 P 458a0c68846640e7b375f7500abe9076 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cd595f79664a402192976b9a664b71e7 (127.17.53.67:46647), 9bfa2167798b44b1aeb4934b00d01783 (127.17.53.65:46815)
I20260929 20:23:15.797736 25105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:15.789431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.53.66:39311 (local address 127.17.53.67:46647)
0929 20:23:15.789610 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0929 20:23:15.789615 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:23:15.790124 (+ 509us) server_negotiation.cc:407] Connection header received
0929 20:23:15.790489 (+ 365us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:23:15.790495 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:23:15.790597 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:23:15.790652 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:23:15.791436 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:15.792521 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:15.795225 (+ 2704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:15.796501 (+ 1276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:15.796663 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:15.796737 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0929 20:23:15.797369 (+ 632us) server_negotiation.cc:299] Negotiation successful
0929 20:23:15.797496 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 20:23:15.797778 25107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:15.790306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.53.66:47179 (local address 127.17.53.65:46815)
0929 20:23:15.790531 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0929 20:23:15.790538 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:23:15.790838 (+ 300us) server_negotiation.cc:407] Connection header received
0929 20:23:15.791019 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:23:15.791026 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:23:15.791116 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:23:15.791212 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:23:15.791893 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:15.792846 (+ 953us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:15.795225 (+ 2379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:15.796501 (+ 1276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:15.796666 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:15.796736 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0929 20:23:15.797369 (+ 633us) server_negotiation.cc:299] Negotiation successful
0929 20:23:15.797497 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260929 20:23:15.798048 25117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:15.789335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.67:46647 (local address 127.17.53.66:39311)
0929 20:23:15.790021 (+ 686us) negotiation.cc:107] Waiting for socket to connect
0929 20:23:15.790042 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 20:23:15.790263 (+ 221us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:23:15.790849 (+ 586us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:23:15.790856 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:23:15.790915 (+ 59us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:23:15.791281 (+ 366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:23:15.791294 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:23:15.792671 (+ 1377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:23:15.792676 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:23:15.795050 (+ 2374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:23:15.795064 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:23:15.796780 (+ 1716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:23:15.796786 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:23:15.797080 (+ 294us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:15.797234 (+ 154us) client_negotiation.cc:769] Sending connection context
0929 20:23:15.797328 (+ 94us) client_negotiation.cc:240] Negotiation successful
0929 20:23:15.797561 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":544,"mutex_wait_us":222,"thread_start_us":349,"threads_started":1}
I20260929 20:23:15.798146 25118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:15.790089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.65:46815 (local address 127.17.53.66:47179)
0929 20:23:15.790673 (+ 584us) negotiation.cc:107] Waiting for socket to connect
0929 20:23:15.790687 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 20:23:15.790884 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:23:15.791273 (+ 389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:23:15.791277 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:23:15.791305 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:23:15.791727 (+ 422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:23:15.791739 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:23:15.792969 (+ 1230us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:23:15.792974 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:23:15.795064 (+ 2090us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:23:15.795072 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:23:15.796780 (+ 1708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:23:15.796786 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:23:15.797079 (+ 293us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:15.797232 (+ 153us) client_negotiation.cc:769] Sending connection context
0929 20:23:15.797328 (+ 96us) client_negotiation.cc:240] Negotiation successful
0929 20:23:15.797561 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":495,"mutex_wait_us":210,"thread_start_us":296,"threads_started":1}
I20260929 20:23:15.799484 24739 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e00ee99c5f7428aa7f5976cc5925293" candidate_uuid: "458a0c68846640e7b375f7500abe9076" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bfa2167798b44b1aeb4934b00d01783" is_pre_election: true
I20260929 20:23:15.799357 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e00ee99c5f7428aa7f5976cc5925293" candidate_uuid: "458a0c68846640e7b375f7500abe9076" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd595f79664a402192976b9a664b71e7" is_pre_election: true
I20260929 20:23:15.800392 24739 raft_consensus.cc:2454] T 0e00ee99c5f7428aa7f5976cc5925293 P 9bfa2167798b44b1aeb4934b00d01783 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 458a0c68846640e7b375f7500abe9076 in term 0.
I20260929 20:23:15.800393 25026 raft_consensus.cc:2454] T 0e00ee99c5f7428aa7f5976cc5925293 P cd595f79664a402192976b9a664b71e7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 458a0c68846640e7b375f7500abe9076 in term 0.
I20260929 20:23:15.801469 24818 leader_election.cc:302] T 0e00ee99c5f7428aa7f5976cc5925293 P 458a0c68846640e7b375f7500abe9076 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 458a0c68846640e7b375f7500abe9076, 9bfa2167798b44b1aeb4934b00d01783; no voters:
I20260929 20:23:15.802006 25114 raft_consensus.cc:2781] T 0e00ee99c5f7428aa7f5976cc5925293 P 458a0c68846640e7b375f7500abe9076 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 20:23:15.802280 25114 raft_consensus.cc:482] T 0e00ee99c5f7428aa7f5976cc5925293 P 458a0c68846640e7b375f7500abe9076 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 20:23:15.802470 25114 raft_consensus.cc:3037] T 0e00ee99c5f7428aa7f5976cc5925293 P 458a0c68846640e7b375f7500abe9076 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:23:15.805605 25114 raft_consensus.cc:504] T 0e00ee99c5f7428aa7f5976cc5925293 P 458a0c68846640e7b375f7500abe9076 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "458a0c68846640e7b375f7500abe9076" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 35805 } } peers { permanent_uuid: "cd595f79664a402192976b9a664b71e7" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 46647 } } peers { permanent_uuid: "9bfa2167798b44b1aeb4934b00d01783" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 46815 } }
I20260929 20:23:15.806361 25114 leader_election.cc:288] T 0e00ee99c5f7428aa7f5976cc5925293 P 458a0c68846640e7b375f7500abe9076 [CANDIDATE]: Term 1 election: Requested vote from peers cd595f79664a402192976b9a664b71e7 (127.17.53.67:46647), 9bfa2167798b44b1aeb4934b00d01783 (127.17.53.65:46815)
I20260929 20:23:15.806787 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e00ee99c5f7428aa7f5976cc5925293" candidate_uuid: "458a0c68846640e7b375f7500abe9076" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd595f79664a402192976b9a664b71e7"
I20260929 20:23:15.806841 24739 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e00ee99c5f7428aa7f5976cc5925293" candidate_uuid: "458a0c68846640e7b375f7500abe9076" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bfa2167798b44b1aeb4934b00d01783"
I20260929 20:23:15.807090 24739 raft_consensus.cc:3037] T 0e00ee99c5f7428aa7f5976cc5925293 P 9bfa2167798b44b1aeb4934b00d01783 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:23:15.807104 25026 raft_consensus.cc:3037] T 0e00ee99c5f7428aa7f5976cc5925293 P cd595f79664a402192976b9a664b71e7 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:23:15.810259 24739 raft_consensus.cc:2454] T 0e00ee99c5f7428aa7f5976cc5925293 P 9bfa2167798b44b1aeb4934b00d01783 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 458a0c68846640e7b375f7500abe9076 in term 1.
I20260929 20:23:15.810217 25026 raft_consensus.cc:2454] T 0e00ee99c5f7428aa7f5976cc5925293 P cd595f79664a402192976b9a664b71e7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 458a0c68846640e7b375f7500abe9076 in term 1.
I20260929 20:23:15.810770 24818 leader_election.cc:302] T 0e00ee99c5f7428aa7f5976cc5925293 P 458a0c68846640e7b375f7500abe9076 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 458a0c68846640e7b375f7500abe9076, cd595f79664a402192976b9a664b71e7; no voters:
I20260929 20:23:15.811185 25114 raft_consensus.cc:2781] T 0e00ee99c5f7428aa7f5976cc5925293 P 458a0c68846640e7b375f7500abe9076 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:23:15.812101 25114 raft_consensus.cc:686] T 0e00ee99c5f7428aa7f5976cc5925293 P 458a0c68846640e7b375f7500abe9076 [term 1 LEADER]: Becoming Leader. State: Replica: 458a0c68846640e7b375f7500abe9076, State: Running, Role: LEADER
I20260929 20:23:15.812829 25114 consensus_queue.cc:237] T 0e00ee99c5f7428aa7f5976cc5925293 P 458a0c68846640e7b375f7500abe9076 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "458a0c68846640e7b375f7500abe9076" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 35805 } } peers { permanent_uuid: "cd595f79664a402192976b9a664b71e7" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 46647 } } peers { permanent_uuid: "9bfa2167798b44b1aeb4934b00d01783" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 46815 } }
I20260929 20:23:15.818501 23816 catalog_manager.cc:5495] T 0e00ee99c5f7428aa7f5976cc5925293 P 458a0c68846640e7b375f7500abe9076 reported cstate change: term changed from 0 to 1, leader changed from <none> to 458a0c68846640e7b375f7500abe9076 (127.17.53.66). New cstate: current_term: 1 leader_uuid: "458a0c68846640e7b375f7500abe9076" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "458a0c68846640e7b375f7500abe9076" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 35805 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "cd595f79664a402192976b9a664b71e7" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 46647 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9bfa2167798b44b1aeb4934b00d01783" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 46815 } health_report { overall_health: UNKNOWN } } }
I20260929 20:23:15.850180 25103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:15.844510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43268 (local address 127.17.53.66:35805)
0929 20:23:15.844750 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0929 20:23:15.844754 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:23:15.844770 (+ 16us) server_negotiation.cc:407] Connection header received
0929 20:23:15.844851 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:23:15.844854 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:23:15.844936 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:23:15.845080 (+ 144us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 20:23:15.845471 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:15.846165 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:15.847496 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:15.848163 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:15.848287 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:15.848841 (+ 554us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 20:23:15.849561 (+ 720us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 20:23:15.849701 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0929 20:23:15.849810 (+ 109us) server_negotiation.cc:299] Negotiation successful
0929 20:23:15.849902 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121}
I20260929 20:23:15.859324 25026 raft_consensus.cc:1261] T 0e00ee99c5f7428aa7f5976cc5925293 P cd595f79664a402192976b9a664b71e7 [term 1 FOLLOWER]: Refusing update from remote peer 458a0c68846640e7b375f7500abe9076: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:23:15.859565 24739 raft_consensus.cc:1261] T 0e00ee99c5f7428aa7f5976cc5925293 P 9bfa2167798b44b1aeb4934b00d01783 [term 1 FOLLOWER]: Refusing update from remote peer 458a0c68846640e7b375f7500abe9076: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:23:15.860379 25119 consensus_queue.cc:1035] T 0e00ee99c5f7428aa7f5976cc5925293 P 458a0c68846640e7b375f7500abe9076 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd595f79664a402192976b9a664b71e7" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 46647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:23:15.860821 25114 consensus_queue.cc:1035] T 0e00ee99c5f7428aa7f5976cc5925293 P 458a0c68846640e7b375f7500abe9076 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9bfa2167798b44b1aeb4934b00d01783" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 46815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:23:15.874760 25122 mvcc.cc:205] Tried to move back new op lower bound from 7334762069429657600 to 7334762069253627904. Current Snapshot: MvccSnapshot[applied={T|T < 7334762069429657600}]
I20260929 20:23:15.892795 23817 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59728:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 20:23:15.893105 23817 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59728:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 20:23:15.899868 23817 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 94c779fe1aa149a9add4b5b63f72b2df: Sending DeleteTablet for 3 replicas of tablet 0e00ee99c5f7428aa7f5976cc5925293
I20260929 20:23:15.901144 25006 tablet_service.cc:1514] Processing DeleteTablet for tablet 0e00ee99c5f7428aa7f5976cc5925293 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:23:15 UTC) from {username='oryx', principal='oryx/127.17.53.126@KRBTEST.COM'} at 127.0.0.1:45028
I20260929 20:23:15.901214 24864 tablet_service.cc:1514] Processing DeleteTablet for tablet 0e00ee99c5f7428aa7f5976cc5925293 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:23:15 UTC) from {username='oryx', principal='oryx/127.17.53.126@KRBTEST.COM'} at 127.0.0.1:43262
I20260929 20:23:15.901238 24719 tablet_service.cc:1514] Processing DeleteTablet for tablet 0e00ee99c5f7428aa7f5976cc5925293 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:23:15 UTC) from {username='oryx', principal='oryx/127.17.53.126@KRBTEST.COM'} at 127.0.0.1:45890
I20260929 20:23:15.910964 25132 tablet_replica.cc:330] T 0e00ee99c5f7428aa7f5976cc5925293 P 458a0c68846640e7b375f7500abe9076: stopping tablet replica
I20260929 20:23:15.911867 25132 raft_consensus.cc:2229] T 0e00ee99c5f7428aa7f5976cc5925293 P 458a0c68846640e7b375f7500abe9076 [term 1 LEADER]: Raft consensus shutting down.
I20260929 20:23:15.912878 25132 raft_consensus.cc:2258] T 0e00ee99c5f7428aa7f5976cc5925293 P 458a0c68846640e7b375f7500abe9076 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:23:15.913892 25133 tablet_replica.cc:330] T 0e00ee99c5f7428aa7f5976cc5925293 P 9bfa2167798b44b1aeb4934b00d01783: stopping tablet replica
I20260929 20:23:15.916007 25132 ts_tablet_manager.cc:1905] T 0e00ee99c5f7428aa7f5976cc5925293 P 458a0c68846640e7b375f7500abe9076: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:23:15.916747 17621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu pid 23785
I20260929 20:23:15.917362 25133 raft_consensus.cc:2229] T 0e00ee99c5f7428aa7f5976cc5925293 P 9bfa2167798b44b1aeb4934b00d01783 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 20:23:15.917918 23847 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:23:15.918483 25133 raft_consensus.cc:2258] T 0e00ee99c5f7428aa7f5976cc5925293 P 9bfa2167798b44b1aeb4934b00d01783 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:23:15.921729 25131 tablet_replica.cc:330] T 0e00ee99c5f7428aa7f5976cc5925293 P cd595f79664a402192976b9a664b71e7: stopping tablet replica
I20260929 20:23:15.923235 25133 ts_tablet_manager.cc:1905] T 0e00ee99c5f7428aa7f5976cc5925293 P 9bfa2167798b44b1aeb4934b00d01783: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:23:15.927387 25131 raft_consensus.cc:2229] T 0e00ee99c5f7428aa7f5976cc5925293 P cd595f79664a402192976b9a664b71e7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 20:23:15.927969 25131 raft_consensus.cc:2258] T 0e00ee99c5f7428aa7f5976cc5925293 P cd595f79664a402192976b9a664b71e7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:23:15.930385 25131 ts_tablet_manager.cc:1905] T 0e00ee99c5f7428aa7f5976cc5925293 P cd595f79664a402192976b9a664b71e7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:23:15.931298 25133 ts_tablet_manager.cc:1918] T 0e00ee99c5f7428aa7f5976cc5925293 P 9bfa2167798b44b1aeb4934b00d01783: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 20:23:15.931602 25133 log.cc:1196] T 0e00ee99c5f7428aa7f5976cc5925293 P 9bfa2167798b44b1aeb4934b00d01783: Deleting WAL directory at /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713339161492-17621-0/minicluster-data/ts-0/wal/wals/0e00ee99c5f7428aa7f5976cc5925293
I20260929 20:23:15.932240 25133 ts_tablet_manager.cc:1939] T 0e00ee99c5f7428aa7f5976cc5925293 P 9bfa2167798b44b1aeb4934b00d01783: Deleting consensus metadata
I20260929 20:23:15.933538 25089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:15.905394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48088 (local address 127.17.53.125:35701)
0929 20:23:15.905604 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0929 20:23:15.905614 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 20:23:15.914256 (+ 8642us) server_negotiation.cc:407] Connection header received
0929 20:23:15.914482 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:23:15.914491 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:23:15.914624 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:23:15.914772 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:23:15.918029 (+ 3257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:15.919549 (+ 1520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:15.921735 (+ 2186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:15.922670 (+ 935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:15.922845 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:15.926379 (+ 3534us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:23:15.926389 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:23:15.926397 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:23:15.926454 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:23:15.930419 (+ 3965us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:23:15.931323 (+ 904us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:23:15.931329 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:23:15.931332 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:23:15.931476 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:23:15.931885 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:23:15.931890 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:23:15.931893 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:23:15.932247 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:23:15.932504 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0929 20:23:15.932925 (+ 421us) server_negotiation.cc:299] Negotiation successful
0929 20:23:15.933171 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260929 20:23:15.937022 25132 ts_tablet_manager.cc:1918] T 0e00ee99c5f7428aa7f5976cc5925293 P 458a0c68846640e7b375f7500abe9076: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 20:23:15.937219 25132 log.cc:1196] T 0e00ee99c5f7428aa7f5976cc5925293 P 458a0c68846640e7b375f7500abe9076: Deleting WAL directory at /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713339161492-17621-0/minicluster-data/ts-1/wal/wals/0e00ee99c5f7428aa7f5976cc5925293
I20260929 20:23:15.937673 25131 ts_tablet_manager.cc:1918] T 0e00ee99c5f7428aa7f5976cc5925293 P cd595f79664a402192976b9a664b71e7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 20:23:15.937839 25131 log.cc:1196] T 0e00ee99c5f7428aa7f5976cc5925293 P cd595f79664a402192976b9a664b71e7: Deleting WAL directory at /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713339161492-17621-0/minicluster-data/ts-2/wal/wals/0e00ee99c5f7428aa7f5976cc5925293
I20260929 20:23:15.937847 25132 ts_tablet_manager.cc:1939] T 0e00ee99c5f7428aa7f5976cc5925293 P 458a0c68846640e7b375f7500abe9076: Deleting consensus metadata
I20260929 20:23:15.938334 25131 ts_tablet_manager.cc:1939] T 0e00ee99c5f7428aa7f5976cc5925293 P cd595f79664a402192976b9a664b71e7: Deleting consensus metadata
I20260929 20:23:15.951112 25088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:15.915617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45014 (local address 127.17.53.124:36147)
0929 20:23:15.915810 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0929 20:23:15.915816 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:23:15.934794 (+ 18978us) server_negotiation.cc:407] Connection header received
0929 20:23:15.935108 (+ 314us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:23:15.935114 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:23:15.935235 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:23:15.935306 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:23:15.937281 (+ 1975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:15.938421 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:15.940461 (+ 2040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:15.941116 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:15.941259 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:15.944651 (+ 3392us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:23:15.944658 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:23:15.944663 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:23:15.944701 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:23:15.948060 (+ 3359us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:23:15.948873 (+ 813us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:23:15.948878 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:23:15.948882 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:23:15.949049 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:23:15.949471 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:23:15.949476 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:23:15.949479 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:23:15.949858 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:23:15.950112 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0929 20:23:15.950470 (+ 358us) server_negotiation.cc:299] Negotiation successful
0929 20:23:15.950690 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260929 20:23:16.060899 23802 catalog_manager.cc:4841] TS cd595f79664a402192976b9a664b71e7 (127.17.53.67:46647): tablet 0e00ee99c5f7428aa7f5976cc5925293 (table test-table [id=67639125246d4b2ba5efff4041cd0f45]) successfully deleted
I20260929 20:23:16.061712 23804 catalog_manager.cc:4841] TS 458a0c68846640e7b375f7500abe9076 (127.17.53.66:35805): tablet 0e00ee99c5f7428aa7f5976cc5925293 (table test-table [id=67639125246d4b2ba5efff4041cd0f45]) successfully deleted
I20260929 20:23:16.061969 23802 catalog_manager.cc:4841] TS 9bfa2167798b44b1aeb4934b00d01783 (127.17.53.65:46815): tablet 0e00ee99c5f7428aa7f5976cc5925293 (table test-table [id=67639125246d4b2ba5efff4041cd0f45]) successfully deleted
I20260929 20:23:16.062474 17621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu with pid 23785
I20260929 20:23:16.116578 25089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:16.089427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48102 (local address 127.17.53.125:35701)
0929 20:23:16.089813 (+ 386us) server_negotiation.cc:206] Beginning negotiation
0929 20:23:16.089820 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:23:16.099000 (+ 9180us) server_negotiation.cc:407] Connection header received
0929 20:23:16.099509 (+ 509us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:23:16.099516 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:23:16.099605 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:23:16.099681 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:23:16.101959 (+ 2278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:16.103093 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:16.105112 (+ 2019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:16.105744 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:16.105838 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:16.110310 (+ 4472us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:23:16.110318 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:23:16.110325 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:23:16.110375 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:23:16.113699 (+ 3324us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:23:16.114599 (+ 900us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:23:16.114604 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:23:16.114607 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:23:16.114750 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:23:16.115172 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:23:16.115178 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:23:16.115182 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:23:16.115535 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:23:16.115788 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0929 20:23:16.116132 (+ 344us) server_negotiation.cc:299] Negotiation successful
0929 20:23:16.116292 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":260}
I20260929 20:23:16.124418 25088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:16.089621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45028 (local address 127.17.53.124:36147)
0929 20:23:16.089944 (+ 323us) server_negotiation.cc:206] Beginning negotiation
0929 20:23:16.089949 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:23:16.107275 (+ 17326us) server_negotiation.cc:407] Connection header received
0929 20:23:16.108334 (+ 1059us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:23:16.108340 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:23:16.108419 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:23:16.108531 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:23:16.110256 (+ 1725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:16.111879 (+ 1623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:16.113824 (+ 1945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:16.114471 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:16.114600 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:16.119018 (+ 4418us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:23:16.119022 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:23:16.119026 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:23:16.119059 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:23:16.121847 (+ 2788us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:23:16.122595 (+ 748us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:23:16.122599 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:23:16.122601 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:23:16.122733 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:23:16.123155 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:23:16.123158 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:23:16.123160 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:23:16.123422 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:23:16.123600 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0929 20:23:16.123998 (+ 398us) server_negotiation.cc:299] Negotiation successful
0929 20:23:16.124147 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":216}
I20260929 20:23:16.882788 25143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:16.881774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:34513 (local address 127.17.53.67:59537)
0929 20:23:16.882308 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0929 20:23:16.882529 (+ 221us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.53.126:34513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":394,"thread_start_us":290,"threads_started":1}
W20260929 20:23:16.883317 25072 heartbeater.cc:644] Failed to heartbeat to 127.17.53.126:34513 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.17.53.126:34513: connect: Connection refused (error 111)
I20260929 20:23:16.886193 25144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:16.885284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:34513 (local address 127.17.53.65:49559)
0929 20:23:16.885806 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0929 20:23:16.885986 (+ 180us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.53.126:34513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":404,"thread_start_us":300,"threads_started":1}
I20260929 20:23:16.886518 25145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:16.885453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:34513 (local address 127.17.53.66:53463)
0929 20:23:16.886189 (+ 736us) negotiation.cc:107] Waiting for socket to connect
0929 20:23:16.886339 (+ 150us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.53.126:34513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":625,"thread_start_us":543,"threads_started":1}
W20260929 20:23:16.886803 24785 heartbeater.cc:644] Failed to heartbeat to 127.17.53.126:34513 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.17.53.126:34513: connect: Connection refused (error 111)
W20260929 20:23:16.886848 24930 heartbeater.cc:644] Failed to heartbeat to 127.17.53.126:34513 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.17.53.126:34513: connect: Connection refused (error 111)
I20260929 20:23:17.400617 25146 raft_consensus.cc:482] T 00000000000000000000000000000000 P cde5a4bc3bab4d4489868351ecc30196 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 94c779fe1aa149a9add4b5b63f72b2df)
I20260929 20:23:17.400862 25147 raft_consensus.cc:482] T 00000000000000000000000000000000 P bbbb7fddd6f64d79bb894cf8c495922e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 94c779fe1aa149a9add4b5b63f72b2df)
I20260929 20:23:17.400893 25146 raft_consensus.cc:504] T 00000000000000000000000000000000 P cde5a4bc3bab4d4489868351ecc30196 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "94c779fe1aa149a9add4b5b63f72b2df" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 34513 } } peers { permanent_uuid: "bbbb7fddd6f64d79bb894cf8c495922e" member_type: VOTER last_known_addr { host: "127.17.53.125" port: 35701 } attrs { promote: false } } peers { permanent_uuid: "cde5a4bc3bab4d4489868351ecc30196" member_type: VOTER last_known_addr { host: "127.17.53.124" port: 36147 } attrs { promote: false } }
I20260929 20:23:17.401118 25147 raft_consensus.cc:504] T 00000000000000000000000000000000 P bbbb7fddd6f64d79bb894cf8c495922e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "94c779fe1aa149a9add4b5b63f72b2df" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 34513 } } peers { permanent_uuid: "bbbb7fddd6f64d79bb894cf8c495922e" member_type: VOTER last_known_addr { host: "127.17.53.125" port: 35701 } attrs { promote: false } } peers { permanent_uuid: "cde5a4bc3bab4d4489868351ecc30196" member_type: VOTER last_known_addr { host: "127.17.53.124" port: 36147 } attrs { promote: false } }
I20260929 20:23:17.402249 25146 leader_election.cc:288] T 00000000000000000000000000000000 P cde5a4bc3bab4d4489868351ecc30196 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 94c779fe1aa149a9add4b5b63f72b2df (127.17.53.126:34513), bbbb7fddd6f64d79bb894cf8c495922e (127.17.53.125:35701)
I20260929 20:23:17.402297 25147 leader_election.cc:288] T 00000000000000000000000000000000 P bbbb7fddd6f64d79bb894cf8c495922e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 94c779fe1aa149a9add4b5b63f72b2df (127.17.53.126:34513), cde5a4bc3bab4d4489868351ecc30196 (127.17.53.124:36147)
I20260929 20:23:17.403337 25148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:17.402345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:34513 (local address 127.0.0.1:60106)
0929 20:23:17.402910 (+ 565us) negotiation.cc:107] Waiting for socket to connect
0929 20:23:17.403055 (+ 145us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.53.126:34513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":382,"thread_start_us":212,"threads_started":1}
I20260929 20:23:17.403559 25149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:17.402390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:34513 (local address 127.0.0.1:60114)
0929 20:23:17.403103 (+ 713us) negotiation.cc:107] Waiting for socket to connect
0929 20:23:17.403283 (+ 180us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.53.126:34513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":597,"thread_start_us":459,"threads_started":1}
W20260929 20:23:17.403679 24591 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.53.126:34513: connect: Connection refused (error 111)
W20260929 20:23:17.404747 24402 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.53.126:34513: connect: Connection refused (error 111)
I20260929 20:23:17.413179 25151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:17.412461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:34513 (local address 127.0.0.1:60126)
0929 20:23:17.412737 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0929 20:23:17.412933 (+ 196us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.53.126:34513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116}
I20260929 20:23:17.413183 25148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:17.402695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.125:35701 (local address 127.0.0.1:48114)
0929 20:23:17.403606 (+ 911us) negotiation.cc:107] Waiting for socket to connect
0929 20:23:17.403711 (+ 105us) client_negotiation.cc:174] Beginning negotiation
0929 20:23:17.403946 (+ 235us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:23:17.404685 (+ 739us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:23:17.404690 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:23:17.404720 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:23:17.405041 (+ 321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:23:17.405056 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:23:17.407011 (+ 1955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:23:17.407019 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:23:17.409714 (+ 2695us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:23:17.409734 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:23:17.412084 (+ 2350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:23:17.412090 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:23:17.412398 (+ 308us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:17.412436 (+ 38us) client_negotiation.cc:769] Sending connection context
0929 20:23:17.412536 (+ 100us) client_negotiation.cc:240] Negotiation successful
0929 20:23:17.412722 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":839,"mutex_wait_us":415,"thread_start_us":4339,"threads_started":1}
W20260929 20:23:17.413429 24591 leader_election.cc:334] T 00000000000000000000000000000000 P cde5a4bc3bab4d4489868351ecc30196 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 94c779fe1aa149a9add4b5b63f72b2df (127.17.53.126:34513): Network error: Client connection negotiation failed: client connection to 127.17.53.126:34513: connect: Connection refused (error 111)
I20260929 20:23:17.413249 25150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:17.402922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48114 (local address 127.17.53.125:35701)
0929 20:23:17.403450 (+ 528us) server_negotiation.cc:206] Beginning negotiation
0929 20:23:17.403456 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:23:17.403894 (+ 438us) server_negotiation.cc:407] Connection header received
0929 20:23:17.404126 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:23:17.404134 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:23:17.404262 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:23:17.404371 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:23:17.405309 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:17.406807 (+ 1498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:17.409912 (+ 3105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:17.411916 (+ 2004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:17.412059 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:17.412152 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0929 20:23:17.412708 (+ 556us) server_negotiation.cc:299] Negotiation successful
0929 20:23:17.412836 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":262,"server-negotiator.queue_time_us":410,"thread_start_us":249,"threads_started":1}
I20260929 20:23:17.413285 25152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:17.412590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:34513 (local address 127.0.0.1:60130)
0929 20:23:17.412795 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0929 20:23:17.413020 (+ 225us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.53.126:34513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
W20260929 20:23:17.413936 24402 leader_election.cc:334] T 00000000000000000000000000000000 P bbbb7fddd6f64d79bb894cf8c495922e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 94c779fe1aa149a9add4b5b63f72b2df (127.17.53.126:34513): Network error: Client connection negotiation failed: client connection to 127.17.53.126:34513: connect: Connection refused (error 111)
I20260929 20:23:17.414273 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cde5a4bc3bab4d4489868351ecc30196" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "bbbb7fddd6f64d79bb894cf8c495922e" is_pre_election: true
I20260929 20:23:17.414659 25149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:17.403461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.124:36147 (local address 127.0.0.1:45034)
0929 20:23:17.403762 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0929 20:23:17.403779 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 20:23:17.403889 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:23:17.404802 (+ 913us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:23:17.404810 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:23:17.404845 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:23:17.405168 (+ 323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:23:17.405181 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:23:17.407663 (+ 2482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:23:17.407672 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:23:17.410640 (+ 2968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:23:17.410660 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:23:17.413933 (+ 3273us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:23:17.413938 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:23:17.414111 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:17.414141 (+ 30us) client_negotiation.cc:769] Sending connection context
0929 20:23:17.414205 (+ 64us) client_negotiation.cc:240] Negotiation successful
0929 20:23:17.414346 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":209,"mutex_wait_us":1,"thread_start_us":7228,"threads_started":1}
I20260929 20:23:17.414633 24427 raft_consensus.cc:2454] T 00000000000000000000000000000000 P bbbb7fddd6f64d79bb894cf8c495922e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cde5a4bc3bab4d4489868351ecc30196 in term 1.
I20260929 20:23:17.415186 24593 leader_election.cc:302] T 00000000000000000000000000000000 P cde5a4bc3bab4d4489868351ecc30196 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: bbbb7fddd6f64d79bb894cf8c495922e, cde5a4bc3bab4d4489868351ecc30196; no voters: 94c779fe1aa149a9add4b5b63f72b2df
I20260929 20:23:17.415578 25153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:17.403880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45034 (local address 127.17.53.124:36147)
0929 20:23:17.404432 (+ 552us) server_negotiation.cc:206] Beginning negotiation
0929 20:23:17.404440 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 20:23:17.404460 (+ 20us) server_negotiation.cc:407] Connection header received
0929 20:23:17.404525 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:23:17.404532 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:23:17.404626 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:23:17.405319 (+ 693us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:23:17.405643 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:17.407159 (+ 1516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:17.411086 (+ 3927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:17.413698 (+ 2612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:17.414904 (+ 1206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:17.415022 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0929 20:23:17.415081 (+ 59us) server_negotiation.cc:299] Negotiation successful
0929 20:23:17.415225 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":689,"server-negotiator.queue_time_us":396,"thread_start_us":275,"threads_started":1}
I20260929 20:23:17.415656 25146 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cde5a4bc3bab4d4489868351ecc30196 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260929 20:23:17.415807 25146 raft_consensus.cc:482] T 00000000000000000000000000000000 P cde5a4bc3bab4d4489868351ecc30196 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 94c779fe1aa149a9add4b5b63f72b2df)
I20260929 20:23:17.415913 25146 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cde5a4bc3bab4d4489868351ecc30196 [term 1 FOLLOWER]: Advancing to term 2
I20260929 20:23:17.416153 24616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bbbb7fddd6f64d79bb894cf8c495922e" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "cde5a4bc3bab4d4489868351ecc30196" is_pre_election: true
I20260929 20:23:17.418052 25146 raft_consensus.cc:504] T 00000000000000000000000000000000 P cde5a4bc3bab4d4489868351ecc30196 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "94c779fe1aa149a9add4b5b63f72b2df" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 34513 } } peers { permanent_uuid: "bbbb7fddd6f64d79bb894cf8c495922e" member_type: VOTER last_known_addr { host: "127.17.53.125" port: 35701 } attrs { promote: false } } peers { permanent_uuid: "cde5a4bc3bab4d4489868351ecc30196" member_type: VOTER last_known_addr { host: "127.17.53.124" port: 36147 } attrs { promote: false } }
I20260929 20:23:17.418458 24616 raft_consensus.cc:2379] T 00000000000000000000000000000000 P cde5a4bc3bab4d4489868351ecc30196 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate bbbb7fddd6f64d79bb894cf8c495922e in current term 2: Already voted for candidate cde5a4bc3bab4d4489868351ecc30196 in this term.
I20260929 20:23:17.418715 25146 leader_election.cc:288] T 00000000000000000000000000000000 P cde5a4bc3bab4d4489868351ecc30196 [CANDIDATE]: Term 2 election: Requested vote from peers 94c779fe1aa149a9add4b5b63f72b2df (127.17.53.126:34513), bbbb7fddd6f64d79bb894cf8c495922e (127.17.53.125:35701)
I20260929 20:23:17.419255 25148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:17.418771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:34513 (local address 127.0.0.1:60140)
0929 20:23:17.418971 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0929 20:23:17.419094 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.53.126:34513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112}
I20260929 20:23:17.419222 24427 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cde5a4bc3bab4d4489868351ecc30196" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "bbbb7fddd6f64d79bb894cf8c495922e"
I20260929 20:23:17.419526 24427 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bbbb7fddd6f64d79bb894cf8c495922e [term 1 FOLLOWER]: Advancing to term 2
I20260929 20:23:17.419659 24403 leader_election.cc:302] T 00000000000000000000000000000000 P bbbb7fddd6f64d79bb894cf8c495922e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: bbbb7fddd6f64d79bb894cf8c495922e; no voters: 94c779fe1aa149a9add4b5b63f72b2df, cde5a4bc3bab4d4489868351ecc30196
I20260929 20:23:17.420380 25148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:17.419961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:34513 (local address 127.0.0.1:60156)
0929 20:23:17.420172 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0929 20:23:17.420272 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.53.126:34513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
W20260929 20:23:17.420567 24591 leader_election.cc:334] T 00000000000000000000000000000000 P cde5a4bc3bab4d4489868351ecc30196 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 94c779fe1aa149a9add4b5b63f72b2df (127.17.53.126:34513): Network error: Client connection negotiation failed: client connection to 127.17.53.126:34513: connect: Connection refused (error 111)
I20260929 20:23:17.422080 24427 raft_consensus.cc:2454] T 00000000000000000000000000000000 P bbbb7fddd6f64d79bb894cf8c495922e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cde5a4bc3bab4d4489868351ecc30196 in term 2.
I20260929 20:23:17.422247 25147 raft_consensus.cc:2726] T 00000000000000000000000000000000 P bbbb7fddd6f64d79bb894cf8c495922e [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 20:23:17.422428 24593 leader_election.cc:302] T 00000000000000000000000000000000 P cde5a4bc3bab4d4489868351ecc30196 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: bbbb7fddd6f64d79bb894cf8c495922e, cde5a4bc3bab4d4489868351ecc30196; no voters: 94c779fe1aa149a9add4b5b63f72b2df
I20260929 20:23:17.422704 25146 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cde5a4bc3bab4d4489868351ecc30196 [term 2 FOLLOWER]: Leader election won for term 2
I20260929 20:23:17.423187 25146 raft_consensus.cc:686] T 00000000000000000000000000000000 P cde5a4bc3bab4d4489868351ecc30196 [term 2 LEADER]: Becoming Leader. State: Replica: cde5a4bc3bab4d4489868351ecc30196, State: Running, Role: LEADER
I20260929 20:23:17.423563 25146 consensus_queue.cc:237] T 00000000000000000000000000000000 P cde5a4bc3bab4d4489868351ecc30196 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "94c779fe1aa149a9add4b5b63f72b2df" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 34513 } } peers { permanent_uuid: "bbbb7fddd6f64d79bb894cf8c495922e" member_type: VOTER last_known_addr { host: "127.17.53.125" port: 35701 } attrs { promote: false } } peers { permanent_uuid: "cde5a4bc3bab4d4489868351ecc30196" member_type: VOTER last_known_addr { host: "127.17.53.124" port: 36147 } attrs { promote: false } }
I20260929 20:23:17.425432 25156 sys_catalog.cc:455] T 00000000000000000000000000000000 P cde5a4bc3bab4d4489868351ecc30196 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cde5a4bc3bab4d4489868351ecc30196. Latest consensus state: current_term: 2 leader_uuid: "cde5a4bc3bab4d4489868351ecc30196" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "94c779fe1aa149a9add4b5b63f72b2df" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 34513 } } peers { permanent_uuid: "bbbb7fddd6f64d79bb894cf8c495922e" member_type: VOTER last_known_addr { host: "127.17.53.125" port: 35701 } attrs { promote: false } } peers { permanent_uuid: "cde5a4bc3bab4d4489868351ecc30196" member_type: VOTER last_known_addr { host: "127.17.53.124" port: 36147 } attrs { promote: false } } }
I20260929 20:23:17.425850 25156 sys_catalog.cc:458] T 00000000000000000000000000000000 P cde5a4bc3bab4d4489868351ecc30196 [sys.catalog]: This master's current role is: LEADER
I20260929 20:23:17.426419 25158 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:23:17.430070 25158 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:23:17.430938 25158 catalog_manager.cc:1259] Loaded cluster ID: 8dd81d8e76ec4d3d8e283d6964a8f856
I20260929 20:23:17.431030 25158 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:23:17.432462 25158 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:23:17.433481 25158 catalog_manager.cc:5879] T 00000000000000000000000000000000 P cde5a4bc3bab4d4489868351ecc30196: Loaded TSK: 0
I20260929 20:23:17.434315 25158 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 20:23:17.460528 24606 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45028:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 20:23:17.468365 24606 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 20:23:17.472525 25148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:17.472108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:34513 (local address 127.0.0.1:60172)
0929 20:23:17.472309 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0929 20:23:17.472390 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.53.126:34513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260929 20:23:17.472597 24427 raft_consensus.cc:1261] T 00000000000000000000000000000000 P bbbb7fddd6f64d79bb894cf8c495922e [term 2 FOLLOWER]: Refusing update from remote peer cde5a4bc3bab4d4489868351ecc30196: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260929 20:23:17.473683 25148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:17.473288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:34513 (local address 127.0.0.1:60186)
0929 20:23:17.473452 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0929 20:23:17.473545 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.53.126:34513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260929 20:23:17.473495 25156 consensus_queue.cc:1035] T 00000000000000000000000000000000 P cde5a4bc3bab4d4489868351ecc30196 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bbbb7fddd6f64d79bb894cf8c495922e" member_type: VOTER last_known_addr { host: "127.17.53.125" port: 35701 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20260929 20:23:17.473906 24591 consensus_peers.cc:487] T 00000000000000000000000000000000 P cde5a4bc3bab4d4489868351ecc30196 -> Peer 94c779fe1aa149a9add4b5b63f72b2df (127.17.53.126:34513): Couldn't send request to peer 94c779fe1aa149a9add4b5b63f72b2df. Status: Network error: Client connection negotiation failed: client connection to 127.17.53.126:34513: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260929 20:23:17.476500 25147 sys_catalog.cc:455] T 00000000000000000000000000000000 P bbbb7fddd6f64d79bb894cf8c495922e [sys.catalog]: SysCatalogTable state changed. Reason: New leader cde5a4bc3bab4d4489868351ecc30196. Latest consensus state: current_term: 2 leader_uuid: "cde5a4bc3bab4d4489868351ecc30196" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "94c779fe1aa149a9add4b5b63f72b2df" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 34513 } } peers { permanent_uuid: "bbbb7fddd6f64d79bb894cf8c495922e" member_type: VOTER last_known_addr { host: "127.17.53.125" port: 35701 } attrs { promote: false } } peers { permanent_uuid: "cde5a4bc3bab4d4489868351ecc30196" member_type: VOTER last_known_addr { host: "127.17.53.124" port: 36147 } attrs { promote: false } } }
I20260929 20:23:17.476774 25147 sys_catalog.cc:458] T 00000000000000000000000000000000 P bbbb7fddd6f64d79bb894cf8c495922e [sys.catalog]: This master's current role is: FOLLOWER
I20260929 20:23:17.478329 25156 sys_catalog.cc:455] T 00000000000000000000000000000000 P cde5a4bc3bab4d4489868351ecc30196 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "cde5a4bc3bab4d4489868351ecc30196" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "94c779fe1aa149a9add4b5b63f72b2df" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 34513 } } peers { permanent_uuid: "bbbb7fddd6f64d79bb894cf8c495922e" member_type: VOTER last_known_addr { host: "127.17.53.125" port: 35701 } attrs { promote: false } } peers { permanent_uuid: "cde5a4bc3bab4d4489868351ecc30196" member_type: VOTER last_known_addr { host: "127.17.53.124" port: 36147 } attrs { promote: false } } }
I20260929 20:23:17.478609 25156 sys_catalog.cc:458] T 00000000000000000000000000000000 P cde5a4bc3bab4d4489868351ecc30196 [sys.catalog]: This master's current role is: LEADER
I20260929 20:23:17.479279 25147 sys_catalog.cc:455] T 00000000000000000000000000000000 P bbbb7fddd6f64d79bb894cf8c495922e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "cde5a4bc3bab4d4489868351ecc30196" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "94c779fe1aa149a9add4b5b63f72b2df" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 34513 } } peers { permanent_uuid: "bbbb7fddd6f64d79bb894cf8c495922e" member_type: VOTER last_known_addr { host: "127.17.53.125" port: 35701 } attrs { promote: false } } peers { permanent_uuid: "cde5a4bc3bab4d4489868351ecc30196" member_type: VOTER last_known_addr { host: "127.17.53.124" port: 36147 } attrs { promote: false } } }
I20260929 20:23:17.479560 25147 sys_catalog.cc:458] T 00000000000000000000000000000000 P bbbb7fddd6f64d79bb894cf8c495922e [sys.catalog]: This master's current role is: FOLLOWER
I20260929 20:23:17.482076 25160 mvcc.cc:205] Tried to move back new op lower bound from 7334762076040192000 to 7334762075850158080. Current Snapshot: MvccSnapshot[applied={T|T < 7334762076040192000 or (T in {7334762076040192000})}]
I20260929 20:23:17.502338 25168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:17.488690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45902 (local address 127.17.53.65:46815)
0929 20:23:17.495345 (+ 6655us) server_negotiation.cc:206] Beginning negotiation
0929 20:23:17.495355 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 20:23:17.495380 (+ 25us) server_negotiation.cc:407] Connection header received
0929 20:23:17.495470 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:23:17.495479 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:23:17.495595 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:23:17.495725 (+ 130us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:23:17.496400 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:17.497502 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:17.500174 (+ 2672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:17.501245 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:17.501385 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:17.501496 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0929 20:23:17.501903 (+ 407us) server_negotiation.cc:299] Negotiation successful
0929 20:23:17.502051 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6490,"thread_start_us":6369,"threads_started":1}
I20260929 20:23:17.502661 25151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:17.487431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.66:35805 (local address 127.0.0.1:43284)
0929 20:23:17.487738 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0929 20:23:17.487754 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 20:23:17.487862 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:23:17.496674 (+ 8812us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:23:17.496679 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:23:17.496704 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:23:17.496923 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:23:17.496935 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:23:17.498273 (+ 1338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:23:17.498279 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:23:17.500370 (+ 2091us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:23:17.500388 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:23:17.501802 (+ 1414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:23:17.501808 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:23:17.502003 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:17.502037 (+ 34us) client_negotiation.cc:769] Sending connection context
0929 20:23:17.502140 (+ 103us) client_negotiation.cc:240] Negotiation successful
0929 20:23:17.502284 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":214,"mutex_wait_us":24}
I20260929 20:23:17.503046 25167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:17.488221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.65:46815 (local address 127.0.0.1:45902)
0929 20:23:17.488887 (+ 666us) negotiation.cc:107] Waiting for socket to connect
0929 20:23:17.488910 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0929 20:23:17.489049 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:23:17.495780 (+ 6731us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:23:17.495789 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:23:17.495827 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:23:17.496244 (+ 417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:23:17.496254 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:23:17.497658 (+ 1404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:23:17.497663 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:23:17.499992 (+ 2329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:23:17.500006 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:23:17.501470 (+ 1464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:23:17.501478 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:23:17.501643 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:17.501683 (+ 40us) client_negotiation.cc:769] Sending connection context
0929 20:23:17.501771 (+ 88us) client_negotiation.cc:240] Negotiation successful
0929 20:23:17.501915 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":517,"mutex_wait_us":31,"thread_start_us":786,"threads_started":1}
I20260929 20:23:17.503110 25166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:17.487745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43284 (local address 127.17.53.66:35805)
0929 20:23:17.496290 (+ 8545us) server_negotiation.cc:206] Beginning negotiation
0929 20:23:17.496300 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 20:23:17.496323 (+ 23us) server_negotiation.cc:407] Connection header received
0929 20:23:17.496406 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:23:17.496414 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:23:17.496509 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:23:17.496648 (+ 139us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:23:17.497094 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:17.498106 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:17.500528 (+ 2422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:17.501578 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:17.501702 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:17.501819 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0929 20:23:17.502658 (+ 839us) server_negotiation.cc:299] Negotiation successful
0929 20:23:17.502797 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8388,"thread_start_us":8267,"threads_started":1}
I20260929 20:23:17.503747 25148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:17.487069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.67:46647 (local address 127.0.0.1:45038)
0929 20:23:17.487237 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0929 20:23:17.487257 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 20:23:17.487365 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:23:17.497516 (+ 10151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:23:17.497524 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:23:17.497549 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:23:17.497835 (+ 286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:23:17.497854 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:23:17.499214 (+ 1360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:23:17.499224 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:23:17.500742 (+ 1518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:23:17.500755 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:23:17.502252 (+ 1497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:23:17.502258 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:23:17.502432 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:17.502472 (+ 40us) client_negotiation.cc:769] Sending connection context
0929 20:23:17.502537 (+ 65us) client_negotiation.cc:240] Negotiation successful
0929 20:23:17.502670 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31,"mutex_wait_us":20}
I20260929 20:23:17.503830 24864 tablet_service.cc:1467] Processing CreateTablet for tablet 83149c9aa30244619f47bbdd56535f87 (DEFAULT_TABLE table=test-table [id=6ccfaab647d04717bd2ade8fbd962fe0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:23:17.503883 24719 tablet_service.cc:1467] Processing CreateTablet for tablet 83149c9aa30244619f47bbdd56535f87 (DEFAULT_TABLE table=test-table [id=6ccfaab647d04717bd2ade8fbd962fe0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:23:17.504288 24864 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83149c9aa30244619f47bbdd56535f87. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:23:17.504314 24719 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83149c9aa30244619f47bbdd56535f87. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:23:17.504531 25165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:17.487183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45038 (local address 127.17.53.67:46647)
0929 20:23:17.497143 (+ 9960us) server_negotiation.cc:206] Beginning negotiation
0929 20:23:17.497147 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:23:17.497161 (+ 14us) server_negotiation.cc:407] Connection header received
0929 20:23:17.497266 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:23:17.497273 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:23:17.497360 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:23:17.497998 (+ 638us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:23:17.498178 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:17.499088 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:17.500898 (+ 1810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:17.502091 (+ 1193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:17.503953 (+ 1862us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:17.504055 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0929 20:23:17.504104 (+ 49us) server_negotiation.cc:299] Negotiation successful
0929 20:23:17.504242 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9849,"thread_start_us":9763,"threads_started":1}
I20260929 20:23:17.505316 25006 tablet_service.cc:1467] Processing CreateTablet for tablet 83149c9aa30244619f47bbdd56535f87 (DEFAULT_TABLE table=test-table [id=6ccfaab647d04717bd2ade8fbd962fe0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:23:17.505724 25006 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83149c9aa30244619f47bbdd56535f87. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:23:17.509980 25169 tablet_bootstrap.cc:492] T 83149c9aa30244619f47bbdd56535f87 P 458a0c68846640e7b375f7500abe9076: Bootstrap starting.
I20260929 20:23:17.511394 25170 tablet_bootstrap.cc:492] T 83149c9aa30244619f47bbdd56535f87 P 9bfa2167798b44b1aeb4934b00d01783: Bootstrap starting.
I20260929 20:23:17.511770 25169 tablet_bootstrap.cc:654] T 83149c9aa30244619f47bbdd56535f87 P 458a0c68846640e7b375f7500abe9076: Neither blocks nor log segments found. Creating new log.
I20260929 20:23:17.513550 25172 tablet_bootstrap.cc:492] T 83149c9aa30244619f47bbdd56535f87 P cd595f79664a402192976b9a664b71e7: Bootstrap starting.
I20260929 20:23:17.513702 25170 tablet_bootstrap.cc:654] T 83149c9aa30244619f47bbdd56535f87 P 9bfa2167798b44b1aeb4934b00d01783: Neither blocks nor log segments found. Creating new log.
I20260929 20:23:17.515053 25169 tablet_bootstrap.cc:492] T 83149c9aa30244619f47bbdd56535f87 P 458a0c68846640e7b375f7500abe9076: No bootstrap required, opened a new log
I20260929 20:23:17.515419 25169 ts_tablet_manager.cc:1397] T 83149c9aa30244619f47bbdd56535f87 P 458a0c68846640e7b375f7500abe9076: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260929 20:23:17.515451 25172 tablet_bootstrap.cc:654] T 83149c9aa30244619f47bbdd56535f87 P cd595f79664a402192976b9a664b71e7: Neither blocks nor log segments found. Creating new log.
I20260929 20:23:17.515772 25170 tablet_bootstrap.cc:492] T 83149c9aa30244619f47bbdd56535f87 P 9bfa2167798b44b1aeb4934b00d01783: No bootstrap required, opened a new log
I20260929 20:23:17.515887 25170 ts_tablet_manager.cc:1397] T 83149c9aa30244619f47bbdd56535f87 P 9bfa2167798b44b1aeb4934b00d01783: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260929 20:23:17.516238 25169 raft_consensus.cc:348] T 83149c9aa30244619f47bbdd56535f87 P 458a0c68846640e7b375f7500abe9076 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd595f79664a402192976b9a664b71e7" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 46647 } } peers { permanent_uuid: "458a0c68846640e7b375f7500abe9076" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 35805 } } peers { permanent_uuid: "9bfa2167798b44b1aeb4934b00d01783" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 46815 } }
I20260929 20:23:17.516466 25169 raft_consensus.cc:374] T 83149c9aa30244619f47bbdd56535f87 P 458a0c68846640e7b375f7500abe9076 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:23:17.516577 25169 raft_consensus.cc:729] T 83149c9aa30244619f47bbdd56535f87 P 458a0c68846640e7b375f7500abe9076 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 458a0c68846640e7b375f7500abe9076, State: Initialized, Role: FOLLOWER
I20260929 20:23:17.516628 25170 raft_consensus.cc:348] T 83149c9aa30244619f47bbdd56535f87 P 9bfa2167798b44b1aeb4934b00d01783 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd595f79664a402192976b9a664b71e7" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 46647 } } peers { permanent_uuid: "458a0c68846640e7b375f7500abe9076" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 35805 } } peers { permanent_uuid: "9bfa2167798b44b1aeb4934b00d01783" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 46815 } }
I20260929 20:23:17.516865 25170 raft_consensus.cc:374] T 83149c9aa30244619f47bbdd56535f87 P 9bfa2167798b44b1aeb4934b00d01783 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:23:17.516954 25170 raft_consensus.cc:729] T 83149c9aa30244619f47bbdd56535f87 P 9bfa2167798b44b1aeb4934b00d01783 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bfa2167798b44b1aeb4934b00d01783, State: Initialized, Role: FOLLOWER
I20260929 20:23:17.516819 25169 consensus_queue.cc:260] T 83149c9aa30244619f47bbdd56535f87 P 458a0c68846640e7b375f7500abe9076 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd595f79664a402192976b9a664b71e7" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 46647 } } peers { permanent_uuid: "458a0c68846640e7b375f7500abe9076" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 35805 } } peers { permanent_uuid: "9bfa2167798b44b1aeb4934b00d01783" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 46815 } }
I20260929 20:23:17.517223 25170 consensus_queue.cc:260] T 83149c9aa30244619f47bbdd56535f87 P 9bfa2167798b44b1aeb4934b00d01783 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd595f79664a402192976b9a664b71e7" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 46647 } } peers { permanent_uuid: "458a0c68846640e7b375f7500abe9076" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 35805 } } peers { permanent_uuid: "9bfa2167798b44b1aeb4934b00d01783" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 46815 } }
I20260929 20:23:17.518253 25170 ts_tablet_manager.cc:1428] T 83149c9aa30244619f47bbdd56535f87 P 9bfa2167798b44b1aeb4934b00d01783: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260929 20:23:17.521535 24784 heartbeater.cc:503] Master 127.17.53.124:36147 was elected leader, sending a full tablet report...
I20260929 20:23:17.523897 25169 ts_tablet_manager.cc:1428] T 83149c9aa30244619f47bbdd56535f87 P 458a0c68846640e7b375f7500abe9076: Time spent starting tablet: real 0.008s user 0.001s sys 0.007s
I20260929 20:23:17.526072 25172 tablet_bootstrap.cc:492] T 83149c9aa30244619f47bbdd56535f87 P cd595f79664a402192976b9a664b71e7: No bootstrap required, opened a new log
I20260929 20:23:17.526247 25172 ts_tablet_manager.cc:1397] T 83149c9aa30244619f47bbdd56535f87 P cd595f79664a402192976b9a664b71e7: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.005s
I20260929 20:23:17.526544 24929 heartbeater.cc:503] Master 127.17.53.124:36147 was elected leader, sending a full tablet report...
I20260929 20:23:17.527031 25172 raft_consensus.cc:348] T 83149c9aa30244619f47bbdd56535f87 P cd595f79664a402192976b9a664b71e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd595f79664a402192976b9a664b71e7" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 46647 } } peers { permanent_uuid: "458a0c68846640e7b375f7500abe9076" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 35805 } } peers { permanent_uuid: "9bfa2167798b44b1aeb4934b00d01783" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 46815 } }
I20260929 20:23:17.527304 25172 raft_consensus.cc:374] T 83149c9aa30244619f47bbdd56535f87 P cd595f79664a402192976b9a664b71e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:23:17.527400 25172 raft_consensus.cc:729] T 83149c9aa30244619f47bbdd56535f87 P cd595f79664a402192976b9a664b71e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd595f79664a402192976b9a664b71e7, State: Initialized, Role: FOLLOWER
I20260929 20:23:17.527655 25172 consensus_queue.cc:260] T 83149c9aa30244619f47bbdd56535f87 P cd595f79664a402192976b9a664b71e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd595f79664a402192976b9a664b71e7" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 46647 } } peers { permanent_uuid: "458a0c68846640e7b375f7500abe9076" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 35805 } } peers { permanent_uuid: "9bfa2167798b44b1aeb4934b00d01783" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 46815 } }
I20260929 20:23:17.534394 25178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:17.524976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:34513 (local address 127.17.53.66:42709)
0929 20:23:17.533982 (+ 9006us) negotiation.cc:107] Waiting for socket to connect
0929 20:23:17.534143 (+ 161us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.53.126:34513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":8847,"thread_start_us":8685,"threads_started":1}
I20260929 20:23:17.534622 25177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:17.518995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:34513 (local address 127.17.53.65:60661)
0929 20:23:17.534207 (+ 15212us) negotiation.cc:107] Waiting for socket to connect
0929 20:23:17.534358 (+ 151us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.53.126:34513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":15073,"threads_started":1}
I20260929 20:23:17.535295 25172 ts_tablet_manager.cc:1428] T 83149c9aa30244619f47bbdd56535f87 P cd595f79664a402192976b9a664b71e7: Time spent starting tablet: real 0.009s user 0.003s sys 0.007s
I20260929 20:23:17.537569 25071 heartbeater.cc:503] Master 127.17.53.124:36147 was elected leader, sending a full tablet report...
I20260929 20:23:17.538547 25178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:17.535072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:34513 (local address 127.17.53.66:47535)
0929 20:23:17.538278 (+ 3206us) negotiation.cc:107] Waiting for socket to connect
0929 20:23:17.538391 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.53.126:34513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":3075,"threads_started":1}
I20260929 20:23:17.539155 25177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:17.538663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:34513 (local address 127.17.53.65:37193)
0929 20:23:17.538820 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0929 20:23:17.538974 (+ 154us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.53.126:34513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260929 20:23:17.540599 25181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:17.535852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:34513 (local address 127.17.53.67:34297)
0929 20:23:17.540324 (+ 4472us) negotiation.cc:107] Waiting for socket to connect
0929 20:23:17.540414 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.53.126:34513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":4369,"thread_start_us":4255,"threads_started":1}
I20260929 20:23:17.541477 25181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:17.541186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:34513 (local address 127.17.53.67:37037)
0929 20:23:17.541313 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0929 20:23:17.541378 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.53.126:34513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260929 20:23:17.596520 25175 raft_consensus.cc:482] T 83149c9aa30244619f47bbdd56535f87 P 458a0c68846640e7b375f7500abe9076 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 20:23:17.596825 25175 raft_consensus.cc:504] T 83149c9aa30244619f47bbdd56535f87 P 458a0c68846640e7b375f7500abe9076 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd595f79664a402192976b9a664b71e7" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 46647 } } peers { permanent_uuid: "458a0c68846640e7b375f7500abe9076" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 35805 } } peers { permanent_uuid: "9bfa2167798b44b1aeb4934b00d01783" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 46815 } }
I20260929 20:23:17.597637 25175 leader_election.cc:288] T 83149c9aa30244619f47bbdd56535f87 P 458a0c68846640e7b375f7500abe9076 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cd595f79664a402192976b9a664b71e7 (127.17.53.67:46647), 9bfa2167798b44b1aeb4934b00d01783 (127.17.53.65:46815)
I20260929 20:23:17.598229 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83149c9aa30244619f47bbdd56535f87" candidate_uuid: "458a0c68846640e7b375f7500abe9076" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd595f79664a402192976b9a664b71e7" is_pre_election: true
I20260929 20:23:17.598290 24739 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83149c9aa30244619f47bbdd56535f87" candidate_uuid: "458a0c68846640e7b375f7500abe9076" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bfa2167798b44b1aeb4934b00d01783" is_pre_election: true
I20260929 20:23:17.598513 25026 raft_consensus.cc:2454] T 83149c9aa30244619f47bbdd56535f87 P cd595f79664a402192976b9a664b71e7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 458a0c68846640e7b375f7500abe9076 in term 0.
I20260929 20:23:17.598584 24739 raft_consensus.cc:2454] T 83149c9aa30244619f47bbdd56535f87 P 9bfa2167798b44b1aeb4934b00d01783 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 458a0c68846640e7b375f7500abe9076 in term 0.
I20260929 20:23:17.599138 24818 leader_election.cc:302] T 83149c9aa30244619f47bbdd56535f87 P 458a0c68846640e7b375f7500abe9076 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 458a0c68846640e7b375f7500abe9076, cd595f79664a402192976b9a664b71e7; no voters:
I20260929 20:23:17.599517 25175 raft_consensus.cc:2781] T 83149c9aa30244619f47bbdd56535f87 P 458a0c68846640e7b375f7500abe9076 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 20:23:17.599681 25175 raft_consensus.cc:482] T 83149c9aa30244619f47bbdd56535f87 P 458a0c68846640e7b375f7500abe9076 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 20:23:17.599771 25175 raft_consensus.cc:3037] T 83149c9aa30244619f47bbdd56535f87 P 458a0c68846640e7b375f7500abe9076 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:23:17.604007 25175 raft_consensus.cc:504] T 83149c9aa30244619f47bbdd56535f87 P 458a0c68846640e7b375f7500abe9076 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd595f79664a402192976b9a664b71e7" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 46647 } } peers { permanent_uuid: "458a0c68846640e7b375f7500abe9076" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 35805 } } peers { permanent_uuid: "9bfa2167798b44b1aeb4934b00d01783" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 46815 } }
I20260929 20:23:17.604492 25175 leader_election.cc:288] T 83149c9aa30244619f47bbdd56535f87 P 458a0c68846640e7b375f7500abe9076 [CANDIDATE]: Term 1 election: Requested vote from peers cd595f79664a402192976b9a664b71e7 (127.17.53.67:46647), 9bfa2167798b44b1aeb4934b00d01783 (127.17.53.65:46815)
I20260929 20:23:17.605099 25026 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83149c9aa30244619f47bbdd56535f87" candidate_uuid: "458a0c68846640e7b375f7500abe9076" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd595f79664a402192976b9a664b71e7"
I20260929 20:23:17.605154 24739 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83149c9aa30244619f47bbdd56535f87" candidate_uuid: "458a0c68846640e7b375f7500abe9076" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9bfa2167798b44b1aeb4934b00d01783"
I20260929 20:23:17.605314 25026 raft_consensus.cc:3037] T 83149c9aa30244619f47bbdd56535f87 P cd595f79664a402192976b9a664b71e7 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:23:17.605425 24739 raft_consensus.cc:3037] T 83149c9aa30244619f47bbdd56535f87 P 9bfa2167798b44b1aeb4934b00d01783 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:23:17.607419 25026 raft_consensus.cc:2454] T 83149c9aa30244619f47bbdd56535f87 P cd595f79664a402192976b9a664b71e7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 458a0c68846640e7b375f7500abe9076 in term 1.
I20260929 20:23:17.607537 24739 raft_consensus.cc:2454] T 83149c9aa30244619f47bbdd56535f87 P 9bfa2167798b44b1aeb4934b00d01783 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 458a0c68846640e7b375f7500abe9076 in term 1.
I20260929 20:23:17.607937 24818 leader_election.cc:302] T 83149c9aa30244619f47bbdd56535f87 P 458a0c68846640e7b375f7500abe9076 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 458a0c68846640e7b375f7500abe9076, cd595f79664a402192976b9a664b71e7; no voters:
I20260929 20:23:17.608258 25175 raft_consensus.cc:2781] T 83149c9aa30244619f47bbdd56535f87 P 458a0c68846640e7b375f7500abe9076 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:23:17.608729 25175 raft_consensus.cc:686] T 83149c9aa30244619f47bbdd56535f87 P 458a0c68846640e7b375f7500abe9076 [term 1 LEADER]: Becoming Leader. State: Replica: 458a0c68846640e7b375f7500abe9076, State: Running, Role: LEADER
I20260929 20:23:17.609017 25175 consensus_queue.cc:237] T 83149c9aa30244619f47bbdd56535f87 P 458a0c68846640e7b375f7500abe9076 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd595f79664a402192976b9a664b71e7" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 46647 } } peers { permanent_uuid: "458a0c68846640e7b375f7500abe9076" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 35805 } } peers { permanent_uuid: "9bfa2167798b44b1aeb4934b00d01783" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 46815 } }
I20260929 20:23:17.610337 25180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:17.609582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:34513 (local address 127.17.53.66:43769)
0929 20:23:17.609799 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0929 20:23:17.610033 (+ 234us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.53.126:34513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260929 20:23:17.612044 24606 catalog_manager.cc:5495] T 83149c9aa30244619f47bbdd56535f87 P 458a0c68846640e7b375f7500abe9076 reported cstate change: term changed from 0 to 1, leader changed from <none> to 458a0c68846640e7b375f7500abe9076 (127.17.53.66). New cstate: current_term: 1 leader_uuid: "458a0c68846640e7b375f7500abe9076" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cd595f79664a402192976b9a664b71e7" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 46647 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "458a0c68846640e7b375f7500abe9076" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 35805 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9bfa2167798b44b1aeb4934b00d01783" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 46815 } health_report { overall_health: UNKNOWN } } }
W20260929 20:23:17.635414 24932 tablet.cc:2369] T 83149c9aa30244619f47bbdd56535f87 P 458a0c68846640e7b375f7500abe9076: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 20:23:17.650066 25166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:17.645329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43298 (local address 127.17.53.66:35805)
0929 20:23:17.645558 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0929 20:23:17.645564 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:23:17.645582 (+ 18us) server_negotiation.cc:407] Connection header received
0929 20:23:17.645763 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:23:17.645768 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:23:17.645843 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:23:17.645936 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 20:23:17.646315 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:17.647096 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:17.648400 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:17.648831 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:17.648964 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:17.649347 (+ 383us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 20:23:17.649455 (+ 108us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 20:23:17.649556 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0929 20:23:17.649721 (+ 165us) server_negotiation.cc:299] Negotiation successful
0929 20:23:17.649798 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260929 20:23:17.653069 24739 raft_consensus.cc:1261] T 83149c9aa30244619f47bbdd56535f87 P 9bfa2167798b44b1aeb4934b00d01783 [term 1 FOLLOWER]: Refusing update from remote peer 458a0c68846640e7b375f7500abe9076: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:23:17.653213 25026 raft_consensus.cc:1261] T 83149c9aa30244619f47bbdd56535f87 P cd595f79664a402192976b9a664b71e7 [term 1 FOLLOWER]: Refusing update from remote peer 458a0c68846640e7b375f7500abe9076: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:23:17.653729 25182 consensus_queue.cc:1035] T 83149c9aa30244619f47bbdd56535f87 P 458a0c68846640e7b375f7500abe9076 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9bfa2167798b44b1aeb4934b00d01783" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 46815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:23:17.654047 25175 consensus_queue.cc:1035] T 83149c9aa30244619f47bbdd56535f87 P 458a0c68846640e7b375f7500abe9076 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd595f79664a402192976b9a664b71e7" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 46647 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:23:17.656677 25181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:17.656073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:34513 (local address 127.17.53.67:53541)
0929 20:23:17.656312 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0929 20:23:17.656433 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.53.126:34513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138}
I20260929 20:23:17.656745 25177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:17.656183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:34513 (local address 127.17.53.65:43895)
0929 20:23:17.656405 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0929 20:23:17.656507 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.53.126:34513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118}
I20260929 20:23:17.663763 25181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:17.663327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:34513 (local address 127.17.53.67:49397)
0929 20:23:17.663535 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0929 20:23:17.663626 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.53.126:34513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113}
I20260929 20:23:17.664207 25177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:17.663773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:34513 (local address 127.17.53.65:41451)
0929 20:23:17.663957 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0929 20:23:17.664048 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.53.126:34513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260929 20:23:17.666111 25186 mvcc.cc:205] Tried to move back new op lower bound from 7334762076780929024 to 7334762076609400832. Current Snapshot: MvccSnapshot[applied={T|T < 7334762076780929024 or (T in {7334762076780929024})}]
I20260929 20:23:17.667028 25187 mvcc.cc:205] Tried to move back new op lower bound from 7334762076780929024 to 7334762076609400832. Current Snapshot: MvccSnapshot[applied={T|T < 7334762076780929024}]
I20260929 20:23:17.673341 25180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:17.672870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:34513 (local address 127.17.53.66:60757)
0929 20:23:17.673070 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0929 20:23:17.673156 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.53.126:34513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100}
I20260929 20:23:17.674702 25180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:17.674076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:34513 (local address 127.17.53.66:60801)
0929 20:23:17.674374 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0929 20:23:17.674458 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.53.126:34513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180}
I20260929 20:23:17.677691 24606 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45028:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 20:23:17.678014 24606 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45028:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 20:23:17.683318 24606 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cde5a4bc3bab4d4489868351ecc30196: Sending DeleteTablet for 3 replicas of tablet 83149c9aa30244619f47bbdd56535f87
I20260929 20:23:17.684360 25006 tablet_service.cc:1514] Processing DeleteTablet for tablet 83149c9aa30244619f47bbdd56535f87 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:23:17 UTC) from {username='oryx', principal='oryx/127.17.53.124@KRBTEST.COM'} at 127.0.0.1:45038
I20260929 20:23:17.684659 24719 tablet_service.cc:1514] Processing DeleteTablet for tablet 83149c9aa30244619f47bbdd56535f87 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:23:17 UTC) from {username='oryx', principal='oryx/127.17.53.124@KRBTEST.COM'} at 127.0.0.1:45902
I20260929 20:23:17.685133 24864 tablet_service.cc:1514] Processing DeleteTablet for tablet 83149c9aa30244619f47bbdd56535f87 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:23:17 UTC) from {username='oryx', principal='oryx/127.17.53.124@KRBTEST.COM'} at 127.0.0.1:43284
I20260929 20:23:17.688651 17621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu pid 24586
I20260929 20:23:17.691957 25196 tablet_replica.cc:330] T 83149c9aa30244619f47bbdd56535f87 P 458a0c68846640e7b375f7500abe9076: stopping tablet replica
I20260929 20:23:17.692097 25195 tablet_replica.cc:330] T 83149c9aa30244619f47bbdd56535f87 P 9bfa2167798b44b1aeb4934b00d01783: stopping tablet replica
I20260929 20:23:17.692178 25196 raft_consensus.cc:2229] T 83149c9aa30244619f47bbdd56535f87 P 458a0c68846640e7b375f7500abe9076 [term 1 LEADER]: Raft consensus shutting down.
I20260929 20:23:17.692310 25195 raft_consensus.cc:2229] T 83149c9aa30244619f47bbdd56535f87 P 9bfa2167798b44b1aeb4934b00d01783 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 20:23:17.692646 25195 raft_consensus.cc:2258] T 83149c9aa30244619f47bbdd56535f87 P 9bfa2167798b44b1aeb4934b00d01783 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:23:17.692646 25196 raft_consensus.cc:2258] T 83149c9aa30244619f47bbdd56535f87 P 458a0c68846640e7b375f7500abe9076 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:23:17.693336 25194 tablet_replica.cc:330] T 83149c9aa30244619f47bbdd56535f87 P cd595f79664a402192976b9a664b71e7: stopping tablet replica
I20260929 20:23:17.693545 25194 raft_consensus.cc:2229] T 83149c9aa30244619f47bbdd56535f87 P cd595f79664a402192976b9a664b71e7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 20:23:17.693696 25196 ts_tablet_manager.cc:1905] T 83149c9aa30244619f47bbdd56535f87 P 458a0c68846640e7b375f7500abe9076: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:23:17.693840 25194 raft_consensus.cc:2258] T 83149c9aa30244619f47bbdd56535f87 P cd595f79664a402192976b9a664b71e7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:23:17.693907 25195 ts_tablet_manager.cc:1905] T 83149c9aa30244619f47bbdd56535f87 P 9bfa2167798b44b1aeb4934b00d01783: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:23:17.694887 25194 ts_tablet_manager.cc:1905] T 83149c9aa30244619f47bbdd56535f87 P cd595f79664a402192976b9a664b71e7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:23:17.700595 25195 ts_tablet_manager.cc:1918] T 83149c9aa30244619f47bbdd56535f87 P 9bfa2167798b44b1aeb4934b00d01783: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 20:23:17.700759 25195 log.cc:1196] T 83149c9aa30244619f47bbdd56535f87 P 9bfa2167798b44b1aeb4934b00d01783: Deleting WAL directory at /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713339161492-17621-0/minicluster-data/ts-0/wal/wals/83149c9aa30244619f47bbdd56535f87
I20260929 20:23:17.700736 25196 ts_tablet_manager.cc:1918] T 83149c9aa30244619f47bbdd56535f87 P 458a0c68846640e7b375f7500abe9076: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 20:23:17.700870 25196 log.cc:1196] T 83149c9aa30244619f47bbdd56535f87 P 458a0c68846640e7b375f7500abe9076: Deleting WAL directory at /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713339161492-17621-0/minicluster-data/ts-1/wal/wals/83149c9aa30244619f47bbdd56535f87
I20260929 20:23:17.701241 25195 ts_tablet_manager.cc:1939] T 83149c9aa30244619f47bbdd56535f87 P 9bfa2167798b44b1aeb4934b00d01783: Deleting consensus metadata
I20260929 20:23:17.701301 25196 ts_tablet_manager.cc:1939] T 83149c9aa30244619f47bbdd56535f87 P 458a0c68846640e7b375f7500abe9076: Deleting consensus metadata
I20260929 20:23:17.702179 25194 ts_tablet_manager.cc:1918] T 83149c9aa30244619f47bbdd56535f87 P cd595f79664a402192976b9a664b71e7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 20:23:17.702370 25194 log.cc:1196] T 83149c9aa30244619f47bbdd56535f87 P cd595f79664a402192976b9a664b71e7: Deleting WAL directory at /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790713339161492-17621-0/minicluster-data/ts-2/wal/wals/83149c9aa30244619f47bbdd56535f87
I20260929 20:23:17.702430 24593 catalog_manager.cc:4841] TS 458a0c68846640e7b375f7500abe9076 (127.17.53.66:35805): tablet 83149c9aa30244619f47bbdd56535f87 (table test-table [id=6ccfaab647d04717bd2ade8fbd962fe0]) successfully deleted
I20260929 20:23:17.702772 24591 catalog_manager.cc:4841] TS 9bfa2167798b44b1aeb4934b00d01783 (127.17.53.65:46815): tablet 83149c9aa30244619f47bbdd56535f87 (table test-table [id=6ccfaab647d04717bd2ade8fbd962fe0]) successfully deleted
I20260929 20:23:17.702905 25194 ts_tablet_manager.cc:1939] T 83149c9aa30244619f47bbdd56535f87 P cd595f79664a402192976b9a664b71e7: Deleting consensus metadata
I20260929 20:23:17.704015 24591 catalog_manager.cc:4841] TS cd595f79664a402192976b9a664b71e7 (127.17.53.67:46647): tablet 83149c9aa30244619f47bbdd56535f87 (table test-table [id=6ccfaab647d04717bd2ade8fbd962fe0]) successfully deleted
I20260929 20:23:17.709334 25168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:17.689467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45914 (local address 127.17.53.65:46815)
0929 20:23:17.689728 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0929 20:23:17.689738 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 20:23:17.693139 (+ 3401us) server_negotiation.cc:407] Connection header received
0929 20:23:17.694746 (+ 1607us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:23:17.694751 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:23:17.694839 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:23:17.694962 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:23:17.696968 (+ 2006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:17.698102 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:17.699947 (+ 1845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:17.700613 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:17.700704 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:17.704407 (+ 3703us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:23:17.704421 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:23:17.704437 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:23:17.704479 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:23:17.707044 (+ 2565us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:23:17.707644 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:23:17.707650 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:23:17.707653 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:23:17.707778 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:23:17.708098 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:23:17.708101 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:23:17.708103 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:23:17.708453 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:23:17.708664 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0929 20:23:17.708940 (+ 276us) server_negotiation.cc:299] Negotiation successful
0929 20:23:17.709099 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260929 20:23:17.710006 24779 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:23:17.805078 17621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu with pid 24586
I20260929 20:23:17.823364 17621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu pid 24801
I20260929 20:23:17.835943 25166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:17.824102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43302 (local address 127.17.53.66:35805)
0929 20:23:17.824280 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0929 20:23:17.824284 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:23:17.824301 (+ 17us) server_negotiation.cc:407] Connection header received
0929 20:23:17.824467 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:23:17.824471 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:23:17.824542 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:23:17.824627 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:23:17.825846 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:17.826570 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:17.827813 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:17.828218 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:17.828309 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:17.830778 (+ 2469us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:23:17.830791 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:23:17.830807 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:23:17.830834 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:23:17.833587 (+ 2753us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:23:17.834204 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:23:17.834210 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:23:17.834213 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:23:17.834331 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:23:17.834663 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:23:17.834666 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:23:17.834668 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:23:17.835082 (+ 414us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:23:17.835314 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0929 20:23:17.835535 (+ 221us) server_negotiation.cc:299] Negotiation successful
0929 20:23:17.835686 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 20:23:17.836596 24924 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:23:17.886636 25148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:17.886081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:34513 (local address 127.0.0.1:60200)
0929 20:23:17.886390 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0929 20:23:17.886473 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.53.126:34513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158}
I20260929 20:23:17.887709 25148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:17.887356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:34513 (local address 127.0.0.1:60202)
0929 20:23:17.887513 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0929 20:23:17.887581 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.53.126:34513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260929 20:23:17.956422 17621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu with pid 24801
I20260929 20:23:17.979305 17621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu pid 24942
I20260929 20:23:17.992599 25165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:17.980143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45044 (local address 127.17.53.67:46647)
0929 20:23:17.980355 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0929 20:23:17.980363 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 20:23:17.980399 (+ 36us) server_negotiation.cc:407] Connection header received
0929 20:23:17.980599 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:23:17.980605 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:23:17.980687 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:23:17.980791 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:23:17.982096 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:17.982772 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:17.984048 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:17.984478 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:17.984612 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:17.987103 (+ 2491us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:23:17.987126 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:23:17.987142 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:23:17.987181 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:23:17.989812 (+ 2631us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:23:17.990451 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:23:17.990460 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:23:17.990464 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:23:17.990588 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:23:17.991063 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:23:17.991070 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:23:17.991073 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:23:17.991424 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:23:17.991717 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0929 20:23:17.992042 (+ 325us) server_negotiation.cc:299] Negotiation successful
0929 20:23:17.992282 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260929 20:23:17.993225 25066 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:23:18.097139 17621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu with pid 24942
I20260929 20:23:18.115520 17621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu pid 24279
I20260929 20:23:18.116674 24447 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:23:18.233986 17621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu with pid 24279
I20260929 20:23:18.254823 17621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu pid 24381
I20260929 20:23:18.256016 24636 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:23:18.377616 17621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu with pid 24381
2026-09-29T20:23:18Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (9869 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790713339161492-17621-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 20:23:18 dist-test-slave-zbnc krb5kdc[25205](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 20:23:18 dist-test-slave-zbnc krb5kdc[25205](info): set up 2 sockets
Sep 29 20:23:18 dist-test-slave-zbnc krb5kdc[25205](info): commencing operation
krb5kdc: starting...
W20260929 20:23:19.553249 17621 mini_kdc.cc:121] Time spent starting KDC: real 1.127s user 0.001s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790713339161492-17621-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790713339161492-17621-0/krb5kdc/test-user.keytab.
Sep 29 20:23:19 dist-test-slave-zbnc krb5kdc[25205](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713399, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T20:23:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T20:23:19Z Disabled control of system clock
WARNING: no policy specified for oryx/127.17.53.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.17.53.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.17.53.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790713339161492-17621-0/krb5kdc/oryx_127.17.53.126.keytab.
Entry for principal oryx/127.17.53.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790713339161492-17621-0/krb5kdc/oryx_127.17.53.126.keytab.
I20260929 20:23:19.775291 17621 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
/tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790713339161492-17621-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790713339161492-17621-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790713339161492-17621-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790713339161492-17621-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.53.126:36941
--webserver_interface=127.17.53.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790713339161492-17621-0/krb5kdc/oryx_127.17.53.126.keytab
--principal=oryx/127.17.53.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.53.84:43903
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.53.126:36941
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790713339161492-17621-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790713339161492-17621-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790713339161492-17621-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:23:20.100492 25218 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:23:20.100787 25218 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:23:20.100950 25218 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:23:20.108294 25218 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:23:20.108400 25218 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260929 20:23:20.108492 25218 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:23:20.108580 25218 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:23:20.108650 25218 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:23:20.108760 25218 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:23:20.118538 25218 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.53.84:43903
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790713339161492-17621-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790713339161492-17621-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.53.126:36941
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790713339161492-17621-0/krb5kdc/oryx_127.17.53.126.keytab
--principal=oryx/127.17.53.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.53.126:36941
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790713339161492-17621-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.53.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790713339161492-17621-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:23:20.119882 25218 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:23:20.121255 25218 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:23:20.133086 25223 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:23:20.133108 25224 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:23:20.133294 25226 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:23:20.134087 25218 server_base.cc:1034] running on GCE node
I20260929 20:23:20.134970 25218 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:23:20.136437 25218 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:23:20.137650 25218 hybrid_clock.cc:648] HybridClock initialized: now 1790713400137601 us; error 31 us; skew 500 ppm
I20260929 20:23:20.137992 25218 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:23:20 dist-test-slave-zbnc krb5kdc[25205](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713400, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.53.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:23:20.141067 25218 init.cc:377] Logged in from keytab as oryx/127.17.53.126@KRBTEST.COM (short username oryx)
I20260929 20:23:20.142453 25218 webserver.cc:460] Webserver started at http://127.17.53.126:37503/ using document root <none> and password file <none>
I20260929 20:23:20.143230 25218 fs_manager.cc:359] Metadata directory not provided
I20260929 20:23:20.143316 25218 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:23:20.143584 25218 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:23:20.146446 25218 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790713339161492-17621-0/minicluster-data/master-0/data/instance:
uuid: "d27fa57b2ddf4b438a0d3fd3a5271ac6"
format_stamp: "Formatted at 2026-09-29 20:23:20 on dist-test-slave-zbnc"
server_key: "b171c5ee943eeb204b54101f93765600"
server_key_iv: "8ffbb31ae3916c4efed54b2f29e3704c"
server_key_version: "encryptionkey@0"
I20260929 20:23:20.147256 25218 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790713339161492-17621-0/minicluster-data/master-0/wal/instance:
uuid: "d27fa57b2ddf4b438a0d3fd3a5271ac6"
format_stamp: "Formatted at 2026-09-29 20:23:20 on dist-test-slave-zbnc"
server_key: "b171c5ee943eeb204b54101f93765600"
server_key_iv: "8ffbb31ae3916c4efed54b2f29e3704c"
server_key_version: "encryptionkey@0"
I20260929 20:23:20.152129 25218 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260929 20:23:20.155393 25233 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:23:20.156739 25218 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 20:23:20.156903 25218 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790713339161492-17621-0/minicluster-data/master-0/data,/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790713339161492-17621-0/minicluster-data/master-0/wal
uuid: "d27fa57b2ddf4b438a0d3fd3a5271ac6"
format_stamp: "Formatted at 2026-09-29 20:23:20 on dist-test-slave-zbnc"
server_key: "b171c5ee943eeb204b54101f93765600"
server_key_iv: "8ffbb31ae3916c4efed54b2f29e3704c"
server_key_version: "encryptionkey@0"
I20260929 20:23:20.157114 25218 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790713339161492-17621-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790713339161492-17621-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790713339161492-17621-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:23:20.172612 25218 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:23:20.176628 25218 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:23:20.176908 25218 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:23:20.198437 25218 rpc_server.cc:307] RPC server started. Bound to: 127.17.53.126:36941
I20260929 20:23:20.198413 25294 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.53.126:36941 every 8 connection(s)
I20260929 20:23:20.199842 25218 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790713339161492-17621-0/minicluster-data/master-0/data/info.pb
I20260929 20:23:20.203765 25295 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:23:20.205492 17621 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu as pid 25218
I20260929 20:23:20.205698 17621 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790713339161492-17621-0/minicluster-data/master-0/wal/instance
I20260929 20:23:20.206429 17621 external_mini_cluster.cc:1428] Setting key 9b5befc4be14c10a617e3a35b95c7c2a
I20260929 20:23:20.215952 25295 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d27fa57b2ddf4b438a0d3fd3a5271ac6: Bootstrap starting.
I20260929 20:23:20.219954 25295 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d27fa57b2ddf4b438a0d3fd3a5271ac6: Neither blocks nor log segments found. Creating new log.
I20260929 20:23:20.221341 25295 log.cc:824] T 00000000000000000000000000000000 P d27fa57b2ddf4b438a0d3fd3a5271ac6: Log is configured to *not* fsync() on all Append() calls
Sep 29 20:23:20 dist-test-slave-zbnc krb5kdc[25205](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713399, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.17.53.126@KRBTEST.COM
I20260929 20:23:20.224594 25295 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d27fa57b2ddf4b438a0d3fd3a5271ac6: No bootstrap required, opened a new log
I20260929 20:23:20.230423 25298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:20.208488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60952 (local address 127.17.53.126:36941)
0929 20:23:20.209383 (+ 895us) server_negotiation.cc:206] Beginning negotiation
0929 20:23:20.209398 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 20:23:20.209440 (+ 42us) server_negotiation.cc:407] Connection header received
0929 20:23:20.211404 (+ 1964us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:23:20.211431 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:23:20.211932 (+ 501us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:23:20.212330 (+ 398us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:23:20.214216 (+ 1886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:20.215597 (+ 1381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:20.217389 (+ 1792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:20.218102 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:20.218242 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:20.223124 (+ 4882us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:23:20.223133 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:23:20.223148 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:23:20.223183 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:23:20.226239 (+ 3056us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:23:20.227131 (+ 892us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:23:20.227137 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:23:20.227144 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:23:20.227327 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:23:20.227720 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:23:20.227723 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:23:20.227725 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:23:20.228269 (+ 544us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:23:20.228493 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0929 20:23:20.228938 (+ 445us) server_negotiation.cc:299] Negotiation successful
0929 20:23:20.229308 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":657,"thread_start_us":423,"threads_started":1}
I20260929 20:23:20.233551 25295 raft_consensus.cc:348] T 00000000000000000000000000000000 P d27fa57b2ddf4b438a0d3fd3a5271ac6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d27fa57b2ddf4b438a0d3fd3a5271ac6" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 36941 } }
I20260929 20:23:20.233933 25295 raft_consensus.cc:374] T 00000000000000000000000000000000 P d27fa57b2ddf4b438a0d3fd3a5271ac6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:23:20.234042 25295 raft_consensus.cc:729] T 00000000000000000000000000000000 P d27fa57b2ddf4b438a0d3fd3a5271ac6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d27fa57b2ddf4b438a0d3fd3a5271ac6, State: Initialized, Role: FOLLOWER
I20260929 20:23:20.234576 25295 consensus_queue.cc:260] T 00000000000000000000000000000000 P d27fa57b2ddf4b438a0d3fd3a5271ac6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d27fa57b2ddf4b438a0d3fd3a5271ac6" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 36941 } }
I20260929 20:23:20.234827 25295 raft_consensus.cc:388] T 00000000000000000000000000000000 P d27fa57b2ddf4b438a0d3fd3a5271ac6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 20:23:20.235003 25295 raft_consensus.cc:482] T 00000000000000000000000000000000 P d27fa57b2ddf4b438a0d3fd3a5271ac6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 20:23:20.235175 25295 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d27fa57b2ddf4b438a0d3fd3a5271ac6 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:23:20.237365 25295 raft_consensus.cc:504] T 00000000000000000000000000000000 P d27fa57b2ddf4b438a0d3fd3a5271ac6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d27fa57b2ddf4b438a0d3fd3a5271ac6" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 36941 } }
I20260929 20:23:20.237877 25295 leader_election.cc:302] T 00000000000000000000000000000000 P d27fa57b2ddf4b438a0d3fd3a5271ac6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d27fa57b2ddf4b438a0d3fd3a5271ac6; no voters:
I20260929 20:23:20.238493 25295 leader_election.cc:288] T 00000000000000000000000000000000 P d27fa57b2ddf4b438a0d3fd3a5271ac6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 20:23:20.238633 25300 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d27fa57b2ddf4b438a0d3fd3a5271ac6 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:23:20.240212 25300 raft_consensus.cc:686] T 00000000000000000000000000000000 P d27fa57b2ddf4b438a0d3fd3a5271ac6 [term 1 LEADER]: Becoming Leader. State: Replica: d27fa57b2ddf4b438a0d3fd3a5271ac6, State: Running, Role: LEADER
I20260929 20:23:20.240814 25295 sys_catalog.cc:564] T 00000000000000000000000000000000 P d27fa57b2ddf4b438a0d3fd3a5271ac6 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:23:20.240760 25300 consensus_queue.cc:237] T 00000000000000000000000000000000 P d27fa57b2ddf4b438a0d3fd3a5271ac6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d27fa57b2ddf4b438a0d3fd3a5271ac6" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 36941 } }
I20260929 20:23:20.243661 25302 sys_catalog.cc:455] T 00000000000000000000000000000000 P d27fa57b2ddf4b438a0d3fd3a5271ac6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d27fa57b2ddf4b438a0d3fd3a5271ac6. Latest consensus state: current_term: 1 leader_uuid: "d27fa57b2ddf4b438a0d3fd3a5271ac6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d27fa57b2ddf4b438a0d3fd3a5271ac6" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 36941 } } }
I20260929 20:23:20.243770 25301 sys_catalog.cc:455] T 00000000000000000000000000000000 P d27fa57b2ddf4b438a0d3fd3a5271ac6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d27fa57b2ddf4b438a0d3fd3a5271ac6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d27fa57b2ddf4b438a0d3fd3a5271ac6" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 36941 } } }
I20260929 20:23:20.244031 25302 sys_catalog.cc:458] T 00000000000000000000000000000000 P d27fa57b2ddf4b438a0d3fd3a5271ac6 [sys.catalog]: This master's current role is: LEADER
I20260929 20:23:20.244133 25301 sys_catalog.cc:458] T 00000000000000000000000000000000 P d27fa57b2ddf4b438a0d3fd3a5271ac6 [sys.catalog]: This master's current role is: LEADER
I20260929 20:23:20.248040 25307 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:23:20.253293 25307 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:23:20.262517 25307 catalog_manager.cc:1347] Generated new cluster ID: 2d8acaa196bc4fd3833ed596dfd1926d
I20260929 20:23:20.262642 25307 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:23:20.273944 25307 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 20:23:20.275978 25307 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:23:20.314163 25307 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d27fa57b2ddf4b438a0d3fd3a5271ac6: Generated new TSK 0
I20260929 20:23:20.315255 25307 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.17.53.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.17.53.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.17.53.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790713339161492-17621-0/krb5kdc/oryx_127.17.53.65.keytab.
Entry for principal oryx/127.17.53.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790713339161492-17621-0/krb5kdc/oryx_127.17.53.65.keytab.
I20260929 20:23:20.383249 17621 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
/tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790713339161492-17621-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790713339161492-17621-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790713339161492-17621-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790713339161492-17621-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.53.65:0
--local_ip_for_outbound_sockets=127.17.53.65
--webserver_interface=127.17.53.65
--webserver_port=0
--tserver_master_addrs=127.17.53.126:36941
--keytab_file=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790713339161492-17621-0/krb5kdc/oryx_127.17.53.65.keytab
--principal=oryx/127.17.53.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.53.84:43903
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790713339161492-17621-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790713339161492-17621-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790713339161492-17621-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:23:20.720104 25321 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:23:20.720409 25321 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:23:20.720613 25321 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:23:20.728116 25321 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:23:20.728259 25321 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:23:20.728417 25321 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 20:23:20.728554 25321 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.53.65
I20260929 20:23:20.738387 25321 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.53.84:43903
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790713339161492-17621-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790713339161492-17621-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790713339161492-17621-0/krb5kdc/oryx_127.17.53.65.keytab
--principal=oryx/127.17.53.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.53.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790713339161492-17621-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.17.53.65
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.17.53.126:36941
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.53.65
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790713339161492-17621-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:23:20.739746 25321 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:23:20.741235 25321 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:23:20.755643 25327 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:23:20.755654 25329 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:23:20.756129 25326 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:23:20.756426 25321 server_base.cc:1034] running on GCE node
I20260929 20:23:20.757145 25321 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:23:20.758133 25321 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:23:20.759382 25321 hybrid_clock.cc:648] HybridClock initialized: now 1790713400759315 us; error 47 us; skew 500 ppm
I20260929 20:23:20.759691 25321 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:23:20 dist-test-slave-zbnc krb5kdc[25205](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713400, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.53.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:23:20.763029 25321 init.cc:377] Logged in from keytab as oryx/127.17.53.65@KRBTEST.COM (short username oryx)
I20260929 20:23:20.764485 25321 webserver.cc:460] Webserver started at http://127.17.53.65:43505/ using document root <none> and password file <none>
I20260929 20:23:20.765180 25321 fs_manager.cc:359] Metadata directory not provided
I20260929 20:23:20.765269 25321 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:23:20.765538 25321 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:23:20.768563 25321 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790713339161492-17621-0/minicluster-data/ts-0/data/instance:
uuid: "f3630496a5db48fda15d65cfcfcf175f"
format_stamp: "Formatted at 2026-09-29 20:23:20 on dist-test-slave-zbnc"
server_key: "fc2183edbd4352e8de1eaf58da2659ef"
server_key_iv: "bcdaa6fff736fad47fc850e8f2a5cf97"
server_key_version: "encryptionkey@0"
I20260929 20:23:20.769325 25321 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790713339161492-17621-0/minicluster-data/ts-0/wal/instance:
uuid: "f3630496a5db48fda15d65cfcfcf175f"
format_stamp: "Formatted at 2026-09-29 20:23:20 on dist-test-slave-zbnc"
server_key: "fc2183edbd4352e8de1eaf58da2659ef"
server_key_iv: "bcdaa6fff736fad47fc850e8f2a5cf97"
server_key_version: "encryptionkey@0"
I20260929 20:23:20.774622 25321 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 20:23:20.778258 25336 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:23:20.779714 25321 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 20:23:20.779887 25321 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790713339161492-17621-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790713339161492-17621-0/minicluster-data/ts-0/wal
uuid: "f3630496a5db48fda15d65cfcfcf175f"
format_stamp: "Formatted at 2026-09-29 20:23:20 on dist-test-slave-zbnc"
server_key: "fc2183edbd4352e8de1eaf58da2659ef"
server_key_iv: "bcdaa6fff736fad47fc850e8f2a5cf97"
server_key_version: "encryptionkey@0"
I20260929 20:23:20.780119 25321 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790713339161492-17621-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790713339161492-17621-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790713339161492-17621-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:23:20.797859 25321 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:23:20.802034 25321 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:23:20.802328 25321 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:23:20.805014 25321 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:23:20.805095 25321 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:23:20.805330 25321 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:23:20.805410 25321 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 20:23:20 dist-test-slave-zbnc krb5kdc[25205](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713400, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.53.65@KRBTEST.COM for oryx/127.17.53.126@KRBTEST.COM
I20260929 20:23:20.831427 25345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:20.805804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.53.65:49303 (local address 127.17.53.126:36941)
0929 20:23:20.806368 (+ 564us) server_negotiation.cc:206] Beginning negotiation
0929 20:23:20.806375 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:23:20.807650 (+ 1275us) server_negotiation.cc:407] Connection header received
0929 20:23:20.809757 (+ 2107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:23:20.809767 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:23:20.809871 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:23:20.810007 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:23:20.812329 (+ 2322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:20.813457 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:20.817064 (+ 3607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:20.817775 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:20.817948 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:20.823315 (+ 5367us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:23:20.823321 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:23:20.823326 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:23:20.823372 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:23:20.827265 (+ 3893us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:23:20.828516 (+ 1251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:23:20.828526 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:23:20.828532 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:23:20.828694 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:23:20.829355 (+ 661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:23:20.829358 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:23:20.829360 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:23:20.829648 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:23:20.829757 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0929 20:23:20.830966 (+ 1209us) server_negotiation.cc:299] Negotiation successful
0929 20:23:20.831164 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":444,"thread_start_us":335,"threads_started":1}
I20260929 20:23:20.833547 25346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:20.806539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:36941 (local address 127.17.53.65:49303)
0929 20:23:20.807446 (+ 907us) negotiation.cc:107] Waiting for socket to connect
0929 20:23:20.807505 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0929 20:23:20.809441 (+ 1936us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:23:20.810289 (+ 848us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:23:20.810297 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:23:20.811074 (+ 777us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:23:20.812081 (+ 1007us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:23:20.812116 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:23:20.813707 (+ 1591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:23:20.813716 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:23:20.816840 (+ 3124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:23:20.816859 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:23:20.818093 (+ 1234us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:23:20.818104 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:23:20.818499 (+ 395us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:20.819476 (+ 977us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:23:20.819503 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:23:20.822963 (+ 3460us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:23:20.827456 (+ 4493us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:23:20.827472 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:23:20.827501 (+ 29us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:23:20.828362 (+ 861us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:23:20.828842 (+ 480us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:23:20.828850 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:23:20.828856 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:23:20.829209 (+ 353us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:23:20.829780 (+ 571us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:23:20.829793 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:23:20.830413 (+ 620us) client_negotiation.cc:769] Sending connection context
0929 20:23:20.830804 (+ 391us) client_negotiation.cc:240] Negotiation successful
0929 20:23:20.831586 (+ 782us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":616,"thread_start_us":413,"threads_started":1}
I20260929 20:23:20.856568 25345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:20.842518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.53.65:33853 (local address 127.17.53.126:36941)
0929 20:23:20.842744 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0929 20:23:20.842753 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 20:23:20.843657 (+ 904us) server_negotiation.cc:407] Connection header received
0929 20:23:20.843964 (+ 307us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:23:20.843973 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:23:20.844070 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:23:20.844180 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:23:20.845180 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:20.846100 (+ 920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:20.847676 (+ 1576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:20.848071 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:20.848207 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:20.850107 (+ 1900us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:23:20.850111 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:23:20.850115 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:23:20.850144 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:23:20.853344 (+ 3200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:23:20.854298 (+ 954us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:23:20.854303 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:23:20.854307 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:23:20.854442 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:23:20.854944 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:23:20.854950 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:23:20.854955 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:23:20.855285 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:23:20.855442 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0929 20:23:20.855967 (+ 525us) server_negotiation.cc:299] Negotiation successful
0929 20:23:20.856199 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260929 20:23:20.856751 25428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:20.842694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:36941 (local address 127.17.53.65:33853)
0929 20:23:20.843527 (+ 833us) negotiation.cc:107] Waiting for socket to connect
0929 20:23:20.843557 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0929 20:23:20.843794 (+ 237us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:23:20.844207 (+ 413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:23:20.844211 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:23:20.844705 (+ 494us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:23:20.845028 (+ 323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:23:20.845041 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:23:20.846227 (+ 1186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:23:20.846239 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:23:20.847528 (+ 1289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:23:20.847540 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:23:20.848261 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:23:20.848267 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:23:20.848431 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:20.849150 (+ 719us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:23:20.849166 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:23:20.849915 (+ 749us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:23:20.853527 (+ 3612us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:23:20.853534 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:23:20.853540 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:23:20.854133 (+ 593us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:23:20.854571 (+ 438us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:23:20.854577 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:23:20.854583 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:23:20.854770 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:23:20.855513 (+ 743us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:23:20.855525 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:23:20.855707 (+ 182us) client_negotiation.cc:769] Sending connection context
0929 20:23:20.855875 (+ 168us) client_negotiation.cc:240] Negotiation successful
0929 20:23:20.856230 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":636,"thread_start_us":433,"threads_started":1}
I20260929 20:23:20.858245 25321 rpc_server.cc:307] RPC server started. Bound to: 127.17.53.65:41053
I20260929 20:23:20.858250 25456 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.53.65:41053 every 8 connection(s)
I20260929 20:23:20.859601 25321 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790713339161492-17621-0/minicluster-data/ts-0/data/info.pb
I20260929 20:23:20.861855 25457 heartbeater.cc:347] Connected to a master server at 127.17.53.126:36941
I20260929 20:23:20.862103 25457 heartbeater.cc:464] Registering TS with master...
I20260929 20:23:20.862833 25457 heartbeater.cc:511] Master 127.17.53.126:36941 requested a full tablet report, sending...
I20260929 20:23:20.864836 25250 ts_manager.cc:194] Registered new tserver with Master: f3630496a5db48fda15d65cfcfcf175f (127.17.53.65:41053)
I20260929 20:23:20.866297 25250 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.17.53.65@KRBTEST.COM'} at 127.17.53.65:49303
I20260929 20:23:20.868115 17621 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu as pid 25321
I20260929 20:23:20.868316 17621 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790713339161492-17621-0/minicluster-data/ts-0/wal/instance
I20260929 20:23:20.869103 17621 external_mini_cluster.cc:1428] Setting key d60ba9c7976978c2f4348572f00c73c5
WARNING: no policy specified for oryx/127.17.53.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.17.53.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.17.53.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790713339161492-17621-0/krb5kdc/oryx_127.17.53.66.keytab.
Entry for principal oryx/127.17.53.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790713339161492-17621-0/krb5kdc/oryx_127.17.53.66.keytab.
I20260929 20:23:20.918550 17621 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
/tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790713339161492-17621-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790713339161492-17621-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790713339161492-17621-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790713339161492-17621-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.53.66:0
--local_ip_for_outbound_sockets=127.17.53.66
--webserver_interface=127.17.53.66
--webserver_port=0
--tserver_master_addrs=127.17.53.126:36941
--keytab_file=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790713339161492-17621-0/krb5kdc/oryx_127.17.53.66.keytab
--principal=oryx/127.17.53.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.53.84:43903
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790713339161492-17621-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790713339161492-17621-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790713339161492-17621-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:23:21.264904 25461 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:23:21.265172 25461 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:23:21.265326 25461 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:23:21.272693 25461 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:23:21.272800 25461 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:23:21.272970 25461 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 20:23:21.273092 25461 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.53.66
I20260929 20:23:21.283061 25461 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.53.84:43903
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790713339161492-17621-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790713339161492-17621-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790713339161492-17621-0/krb5kdc/oryx_127.17.53.66.keytab
--principal=oryx/127.17.53.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.53.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790713339161492-17621-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.17.53.66
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.17.53.126:36941
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.53.66
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790713339161492-17621-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:23:21.284432 25461 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:23:21.285811 25461 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:23:21.299398 25469 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:23:21.299445 25467 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:23:21.299360 25466 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:23:21.304646 25461 server_base.cc:1034] running on GCE node
I20260929 20:23:21.305361 25461 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:23:21.306354 25461 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:23:21.307600 25461 hybrid_clock.cc:648] HybridClock initialized: now 1790713401307517 us; error 39 us; skew 500 ppm
I20260929 20:23:21.307917 25461 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:23:21 dist-test-slave-zbnc krb5kdc[25205](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713401, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.53.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:23:21.311240 25461 init.cc:377] Logged in from keytab as oryx/127.17.53.66@KRBTEST.COM (short username oryx)
I20260929 20:23:21.312762 25461 webserver.cc:460] Webserver started at http://127.17.53.66:40199/ using document root <none> and password file <none>
I20260929 20:23:21.313514 25461 fs_manager.cc:359] Metadata directory not provided
I20260929 20:23:21.313648 25461 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:23:21.313989 25461 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:23:21.316979 25461 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790713339161492-17621-0/minicluster-data/ts-1/data/instance:
uuid: "70e535e06c4b430c9ec90f98dbc4f69b"
format_stamp: "Formatted at 2026-09-29 20:23:21 on dist-test-slave-zbnc"
server_key: "3f428622088cb80d3c1bb2983a2dcdc2"
server_key_iv: "e0600d05c7ea849b2edec5b29e0a6b5d"
server_key_version: "encryptionkey@0"
I20260929 20:23:21.317839 25461 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790713339161492-17621-0/minicluster-data/ts-1/wal/instance:
uuid: "70e535e06c4b430c9ec90f98dbc4f69b"
format_stamp: "Formatted at 2026-09-29 20:23:21 on dist-test-slave-zbnc"
server_key: "3f428622088cb80d3c1bb2983a2dcdc2"
server_key_iv: "e0600d05c7ea849b2edec5b29e0a6b5d"
server_key_version: "encryptionkey@0"
I20260929 20:23:21.322909 25461 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260929 20:23:21.326223 25476 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:23:21.327575 25461 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260929 20:23:21.327764 25461 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790713339161492-17621-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790713339161492-17621-0/minicluster-data/ts-1/wal
uuid: "70e535e06c4b430c9ec90f98dbc4f69b"
format_stamp: "Formatted at 2026-09-29 20:23:21 on dist-test-slave-zbnc"
server_key: "3f428622088cb80d3c1bb2983a2dcdc2"
server_key_iv: "e0600d05c7ea849b2edec5b29e0a6b5d"
server_key_version: "encryptionkey@0"
I20260929 20:23:21.328045 25461 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790713339161492-17621-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790713339161492-17621-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790713339161492-17621-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:23:21.353971 25461 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:23:21.358682 25461 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:23:21.359000 25461 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:23:21.361953 25461 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:23:21.362051 25461 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:23:21.362198 25461 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:23:21.362282 25461 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 20:23:21 dist-test-slave-zbnc krb5kdc[25205](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713401, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.53.66@KRBTEST.COM for oryx/127.17.53.126@KRBTEST.COM
I20260929 20:23:21.387342 25485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:21.362537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.53.66:50523 (local address 127.17.53.126:36941)
0929 20:23:21.363253 (+ 716us) server_negotiation.cc:206] Beginning negotiation
0929 20:23:21.363258 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:23:21.364710 (+ 1452us) server_negotiation.cc:407] Connection header received
0929 20:23:21.367286 (+ 2576us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:23:21.367295 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:23:21.367381 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:23:21.367448 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:23:21.369331 (+ 1883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:21.370723 (+ 1392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:21.373699 (+ 2976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:21.374330 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:21.374468 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:21.379532 (+ 5064us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:23:21.379539 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:23:21.379557 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:23:21.379609 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:23:21.383059 (+ 3450us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:23:21.384081 (+ 1022us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:23:21.384087 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:23:21.384091 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:23:21.384267 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:23:21.384882 (+ 615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:23:21.384887 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:23:21.384891 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:23:21.385241 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:23:21.385397 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0929 20:23:21.386448 (+ 1051us) server_negotiation.cc:299] Negotiation successful
0929 20:23:21.386760 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":602,"thread_start_us":505,"threads_started":1}
I20260929 20:23:21.388335 25486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:21.363695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:36941 (local address 127.17.53.66:50523)
0929 20:23:21.364547 (+ 852us) negotiation.cc:107] Waiting for socket to connect
0929 20:23:21.364584 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0929 20:23:21.366921 (+ 2337us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:23:21.368029 (+ 1108us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:23:21.368037 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:23:21.368565 (+ 528us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:23:21.369138 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:23:21.369155 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:23:21.371033 (+ 1878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:23:21.371040 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:23:21.373536 (+ 2496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:23:21.373552 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:23:21.374582 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:23:21.374590 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:23:21.374817 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:21.375847 (+ 1030us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:23:21.375885 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:23:21.379299 (+ 3414us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:23:21.383211 (+ 3912us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:23:21.383223 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:23:21.383250 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:23:21.383927 (+ 677us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:23:21.384438 (+ 511us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:23:21.384445 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:23:21.384454 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:23:21.384745 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:23:21.385371 (+ 626us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:23:21.385379 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:23:21.385925 (+ 546us) client_negotiation.cc:769] Sending connection context
0929 20:23:21.386301 (+ 376us) client_negotiation.cc:240] Negotiation successful
0929 20:23:21.386965 (+ 664us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":615,"thread_start_us":384,"threads_started":1}
I20260929 20:23:21.410734 25485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:21.395100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.53.66:49719 (local address 127.17.53.126:36941)
0929 20:23:21.395319 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0929 20:23:21.395329 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 20:23:21.396397 (+ 1068us) server_negotiation.cc:407] Connection header received
0929 20:23:21.396637 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:23:21.396643 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:23:21.396719 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:23:21.396816 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:23:21.398341 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:21.399500 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:21.401428 (+ 1928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:21.401851 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:21.402002 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:21.403986 (+ 1984us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:23:21.403995 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:23:21.404003 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:23:21.404058 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:23:21.407251 (+ 3193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:23:21.408129 (+ 878us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:23:21.408134 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:23:21.408138 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:23:21.408269 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:23:21.408768 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:23:21.408774 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:23:21.408778 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:23:21.409243 (+ 465us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:23:21.409393 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0929 20:23:21.410358 (+ 965us) server_negotiation.cc:299] Negotiation successful
0929 20:23:21.410518 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 20:23:21.410746 25558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:21.395475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:36941 (local address 127.17.53.66:49719)
0929 20:23:21.396283 (+ 808us) negotiation.cc:107] Waiting for socket to connect
0929 20:23:21.396306 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0929 20:23:21.396469 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:23:21.396841 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:23:21.396845 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:23:21.397606 (+ 761us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:23:21.398193 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:23:21.398205 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:23:21.399626 (+ 1421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:23:21.399631 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:23:21.401283 (+ 1652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:23:21.401299 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:23:21.402050 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:23:21.402057 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:23:21.402244 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:21.403049 (+ 805us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:23:21.403070 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:23:21.403793 (+ 723us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:23:21.407389 (+ 3596us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:23:21.407398 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:23:21.407402 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:23:21.407959 (+ 557us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:23:21.408438 (+ 479us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:23:21.408445 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:23:21.408450 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:23:21.408641 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:23:21.409457 (+ 816us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:23:21.409461 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:23:21.409624 (+ 163us) client_negotiation.cc:769] Sending connection context
0929 20:23:21.409840 (+ 216us) client_negotiation.cc:240] Negotiation successful
0929 20:23:21.410224 (+ 384us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":635,"thread_start_us":479,"threads_started":1}
I20260929 20:23:21.415925 25461 rpc_server.cc:307] RPC server started. Bound to: 127.17.53.66:40107
I20260929 20:23:21.415934 25596 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.53.66:40107 every 8 connection(s)
I20260929 20:23:21.417234 25461 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790713339161492-17621-0/minicluster-data/ts-1/data/info.pb
I20260929 20:23:21.419448 25597 heartbeater.cc:347] Connected to a master server at 127.17.53.126:36941
I20260929 20:23:21.419826 25597 heartbeater.cc:464] Registering TS with master...
I20260929 20:23:21.419803 17621 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu as pid 25461
I20260929 20:23:21.419971 17621 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790713339161492-17621-0/minicluster-data/ts-1/wal/instance
I20260929 20:23:21.420536 17621 external_mini_cluster.cc:1428] Setting key 1568ac0822a69227163198b21007e7e8
I20260929 20:23:21.420653 25597 heartbeater.cc:511] Master 127.17.53.126:36941 requested a full tablet report, sending...
I20260929 20:23:21.422246 25249 ts_manager.cc:194] Registered new tserver with Master: 70e535e06c4b430c9ec90f98dbc4f69b (127.17.53.66:40107)
I20260929 20:23:21.423238 25249 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.17.53.66@KRBTEST.COM'} at 127.17.53.66:50523
WARNING: no policy specified for oryx/127.17.53.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.17.53.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.17.53.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790713339161492-17621-0/krb5kdc/oryx_127.17.53.67.keytab.
Entry for principal oryx/127.17.53.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790713339161492-17621-0/krb5kdc/oryx_127.17.53.67.keytab.
I20260929 20:23:21.470463 17621 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
/tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790713339161492-17621-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790713339161492-17621-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790713339161492-17621-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790713339161492-17621-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.53.67:0
--local_ip_for_outbound_sockets=127.17.53.67
--webserver_interface=127.17.53.67
--webserver_port=0
--tserver_master_addrs=127.17.53.126:36941
--keytab_file=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790713339161492-17621-0/krb5kdc/oryx_127.17.53.67.keytab
--principal=oryx/127.17.53.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.53.84:43903
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790713339161492-17621-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790713339161492-17621-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790713339161492-17621-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:23:21.800300 25601 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:23:21.800587 25601 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:23:21.800716 25601 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:23:21.807904 25601 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:23:21.808020 25601 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:23:21.808164 25601 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260929 20:23:21.808295 25601 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.53.67
I20260929 20:23:21.818177 25601 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.53.84:43903
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790713339161492-17621-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790713339161492-17621-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790713339161492-17621-0/krb5kdc/oryx_127.17.53.67.keytab
--principal=oryx/127.17.53.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.53.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790713339161492-17621-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.17.53.67
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.17.53.126:36941
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.53.67
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790713339161492-17621-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:23:21.819620 25601 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:23:21.820940 25601 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:23:21.833938 25606 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:23:21.833949 25607 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:23:21.833977 25609 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:23:21.834703 25601 server_base.cc:1034] running on GCE node
I20260929 20:23:21.835326 25601 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:23:21.836284 25601 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:23:21.837495 25601 hybrid_clock.cc:648] HybridClock initialized: now 1790713401837431 us; error 47 us; skew 500 ppm
I20260929 20:23:21.837802 25601 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:23:21 dist-test-slave-zbnc krb5kdc[25205](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713401, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.53.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:23:21.840963 25601 init.cc:377] Logged in from keytab as oryx/127.17.53.67@KRBTEST.COM (short username oryx)
I20260929 20:23:21.842590 25601 webserver.cc:460] Webserver started at http://127.17.53.67:46723/ using document root <none> and password file <none>
I20260929 20:23:21.843370 25601 fs_manager.cc:359] Metadata directory not provided
I20260929 20:23:21.843484 25601 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:23:21.843825 25601 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:23:21.846835 25601 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790713339161492-17621-0/minicluster-data/ts-2/data/instance:
uuid: "f42427cf15e945249ac2061305a6f745"
format_stamp: "Formatted at 2026-09-29 20:23:21 on dist-test-slave-zbnc"
server_key: "a4662b3aaa00f7be0f59b77dc3cac238"
server_key_iv: "a72da0b4cf683d2b87bb5d9b376769d1"
server_key_version: "encryptionkey@0"
I20260929 20:23:21.847687 25601 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790713339161492-17621-0/minicluster-data/ts-2/wal/instance:
uuid: "f42427cf15e945249ac2061305a6f745"
format_stamp: "Formatted at 2026-09-29 20:23:21 on dist-test-slave-zbnc"
server_key: "a4662b3aaa00f7be0f59b77dc3cac238"
server_key_iv: "a72da0b4cf683d2b87bb5d9b376769d1"
server_key_version: "encryptionkey@0"
I20260929 20:23:21.852821 25601 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 20:23:21.856233 25616 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:23:21.857805 25601 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260929 20:23:21.857976 25601 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790713339161492-17621-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790713339161492-17621-0/minicluster-data/ts-2/wal
uuid: "f42427cf15e945249ac2061305a6f745"
format_stamp: "Formatted at 2026-09-29 20:23:21 on dist-test-slave-zbnc"
server_key: "a4662b3aaa00f7be0f59b77dc3cac238"
server_key_iv: "a72da0b4cf683d2b87bb5d9b376769d1"
server_key_version: "encryptionkey@0"
I20260929 20:23:21.858249 25601 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790713339161492-17621-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790713339161492-17621-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790713339161492-17621-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:23:21.869796 25457 heartbeater.cc:503] Master 127.17.53.126:36941 was elected leader, sending a full tablet report...
I20260929 20:23:21.886993 25601 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:23:21.891607 25601 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:23:21.891891 25601 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:23:21.894519 25601 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:23:21.894632 25601 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:23:21.894809 25601 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:23:21.894901 25601 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 20:23:21 dist-test-slave-zbnc krb5kdc[25205](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713401, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.53.67@KRBTEST.COM for oryx/127.17.53.126@KRBTEST.COM
I20260929 20:23:21.920467 25485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:21.895297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.53.67:34805 (local address 127.17.53.126:36941)
0929 20:23:21.895471 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0929 20:23:21.895477 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:23:21.897015 (+ 1538us) server_negotiation.cc:407] Connection header received
0929 20:23:21.899699 (+ 2684us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:23:21.899707 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:23:21.899804 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:23:21.899895 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:23:21.902578 (+ 2683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:21.903937 (+ 1359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:21.906476 (+ 2539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:21.907331 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:21.907497 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:21.912772 (+ 5275us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:23:21.912777 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:23:21.912780 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:23:21.912815 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:23:21.916360 (+ 3545us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:23:21.917482 (+ 1122us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:23:21.917489 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:23:21.917493 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:23:21.917650 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:23:21.918271 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:23:21.918283 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:23:21.918287 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:23:21.918598 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:23:21.918756 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0929 20:23:21.919804 (+ 1048us) server_negotiation.cc:299] Negotiation successful
0929 20:23:21.920080 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 20:23:21.921940 25625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:21.895909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:36941 (local address 127.17.53.67:34805)
0929 20:23:21.896826 (+ 917us) negotiation.cc:107] Waiting for socket to connect
0929 20:23:21.896879 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0929 20:23:21.899307 (+ 2428us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:23:21.900452 (+ 1145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:23:21.900468 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:23:21.901307 (+ 839us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:23:21.902324 (+ 1017us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:23:21.902357 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:23:21.904189 (+ 1832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:23:21.904196 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:23:21.906289 (+ 2093us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:23:21.906311 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:23:21.907601 (+ 1290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:23:21.907615 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:23:21.907979 (+ 364us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:21.908903 (+ 924us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:23:21.908942 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:23:21.912503 (+ 3561us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:23:21.916568 (+ 4065us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:23:21.916580 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:23:21.916606 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:23:21.917354 (+ 748us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:23:21.917820 (+ 466us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:23:21.917828 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:23:21.917833 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:23:21.918142 (+ 309us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:23:21.918746 (+ 604us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:23:21.918755 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:23:21.919313 (+ 558us) client_negotiation.cc:769] Sending connection context
0929 20:23:21.919704 (+ 391us) client_negotiation.cc:240] Negotiation successful
0929 20:23:21.920357 (+ 653us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":644,"thread_start_us":402,"threads_started":1}
I20260929 20:23:21.935941 25601 rpc_server.cc:307] RPC server started. Bound to: 127.17.53.67:38515
I20260929 20:23:21.935956 25734 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.53.67:38515 every 8 connection(s)
I20260929 20:23:21.937440 25601 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790713339161492-17621-0/minicluster-data/ts-2/data/info.pb
I20260929 20:23:21.939929 25735 heartbeater.cc:347] Connected to a master server at 127.17.53.126:36941
I20260929 20:23:21.940199 25735 heartbeater.cc:464] Registering TS with master...
I20260929 20:23:21.941007 25735 heartbeater.cc:511] Master 127.17.53.126:36941 requested a full tablet report, sending...
I20260929 20:23:21.941691 17621 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu as pid 25601
I20260929 20:23:21.941865 17621 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790713339161492-17621-0/minicluster-data/ts-2/wal/instance
I20260929 20:23:21.942435 17621 external_mini_cluster.cc:1428] Setting key 8e4c0110802add9425739d57e9e0e812
I20260929 20:23:21.942716 25249 ts_manager.cc:194] Registered new tserver with Master: f42427cf15e945249ac2061305a6f745 (127.17.53.67:38515)
I20260929 20:23:21.943874 25249 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.17.53.67@KRBTEST.COM'} at 127.17.53.67:34805
I20260929 20:23:21.946099 17621 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 20:23:21.955454 25485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:21.941624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.53.67:50643 (local address 127.17.53.126:36941)
0929 20:23:21.941823 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0929 20:23:21.941847 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0929 20:23:21.942512 (+ 665us) server_negotiation.cc:407] Connection header received
0929 20:23:21.942681 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:23:21.942686 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:23:21.942770 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:23:21.942910 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:23:21.944206 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:21.945209 (+ 1003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:21.946918 (+ 1709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:21.947457 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:21.947596 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:21.949586 (+ 1990us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:23:21.949591 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:23:21.949595 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:23:21.949632 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:23:21.952771 (+ 3139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:23:21.953499 (+ 728us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:23:21.953506 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:23:21.953510 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:23:21.953654 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:23:21.954035 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:23:21.954041 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:23:21.954044 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:23:21.954303 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:23:21.954455 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0929 20:23:21.954839 (+ 384us) server_negotiation.cc:299] Negotiation successful
0929 20:23:21.955073 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260929 20:23:21.955533 25737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:21.941727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:36941 (local address 127.17.53.67:50643)
0929 20:23:21.942400 (+ 673us) negotiation.cc:107] Waiting for socket to connect
0929 20:23:21.942420 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 20:23:21.942553 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:23:21.942957 (+ 404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:23:21.942962 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:23:21.943725 (+ 763us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:23:21.944038 (+ 313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:23:21.944053 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:23:21.945399 (+ 1346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:23:21.945405 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:23:21.946754 (+ 1349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:23:21.946768 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:23:21.947647 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:23:21.947654 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:23:21.947848 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:21.948523 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:23:21.948540 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:23:21.949228 (+ 688us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:23:21.952911 (+ 3683us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:23:21.952917 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:23:21.952922 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:23:21.953368 (+ 446us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:23:21.953776 (+ 408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:23:21.953781 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:23:21.953784 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:23:21.953918 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:23:21.954425 (+ 507us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:23:21.954430 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:23:21.954610 (+ 180us) client_negotiation.cc:769] Sending connection context
0929 20:23:21.954754 (+ 144us) client_negotiation.cc:240] Negotiation successful
0929 20:23:21.955073 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":485,"thread_start_us":346,"threads_started":1}
Sep 29 20:23:21 dist-test-slave-zbnc krb5kdc[25205](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713401, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 29 20:23:21 dist-test-slave-zbnc krb5kdc[25205](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.17.53.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 20:23:21 dist-test-slave-zbnc krb5kdc[25205](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.17.53.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 20:23:22.425956 25597 heartbeater.cc:503] Master 127.17.53.126:36941 was elected leader, sending a full tablet report...
I20260929 20:23:22.947510 25735 heartbeater.cc:503] Master 127.17.53.126:36941 was elected leader, sending a full tablet report...
W20260929 20:23:23.059923 25744 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.065s user 0.007s sys 0.007s
W20260929 20:23:23.061107 25485 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:23:21.989104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57842 (local address 127.17.53.126:36941)
0929 20:23:21.989278 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0929 20:23:21.989283 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:23:21.989441 (+ 158us) server_negotiation.cc:407] Connection header received
0929 20:23:21.989615 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:23:21.989619 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:23:21.989693 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:23:21.989782 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:23:21.991239 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:21.992014 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:21.993208 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:21.993615 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:21.993710 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:23.060817 (+1067107us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57842: BlockingRecv error: recv got EOF from 127.0.0.1:57842 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Sep 29 20:23:23 dist-test-slave-zbnc krb5kdc[25205](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713401, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.17.53.126@KRBTEST.COM
I20260929 20:23:23.082835 25485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:23.065264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57850 (local address 127.17.53.126:36941)
0929 20:23:23.065499 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0929 20:23:23.065507 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 20:23:23.065798 (+ 291us) server_negotiation.cc:407] Connection header received
0929 20:23:23.066015 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:23:23.066020 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:23:23.066093 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:23:23.066197 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:23:23.067482 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:23.068273 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:23.069534 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:23.069976 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:23.070112 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:23.078300 (+ 8188us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:23:23.078306 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:23:23.078309 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:23:23.078346 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:23:23.080781 (+ 2435us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:23:23.081384 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:23:23.081388 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:23:23.081390 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:23:23.081505 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:23:23.081804 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:23:23.081807 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:23:23.081809 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:23:23.082052 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:23:23.082245 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0929 20:23:23.082446 (+ 201us) server_negotiation.cc:299] Negotiation successful
0929 20:23:23.082597 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260929 20:23:23.086973 25249 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57850:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 20:23:23.093818 25249 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 20:23:23.130419 25760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:23.107753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46666 (local address 127.17.53.65:41053)
0929 20:23:23.121380 (+ 13627us) server_negotiation.cc:206] Beginning negotiation
0929 20:23:23.121394 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 20:23:23.122654 (+ 1260us) server_negotiation.cc:407] Connection header received
0929 20:23:23.122794 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:23:23.122801 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:23:23.123078 (+ 277us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:23:23.123276 (+ 198us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:23:23.124462 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:23.125984 (+ 1522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:23.127972 (+ 1988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:23.129267 (+ 1295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:23.129422 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:23.129647 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0929 20:23:23.129909 (+ 262us) server_negotiation.cc:299] Negotiation successful
0929 20:23:23.130076 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13475,"thread_start_us":13314,"threads_started":1}
I20260929 20:23:23.130499 25757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:23.107171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.65:41053 (local address 127.0.0.1:46666)
0929 20:23:23.122432 (+ 15261us) negotiation.cc:107] Waiting for socket to connect
0929 20:23:23.122451 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 20:23:23.122581 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:23:23.123499 (+ 918us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:23:23.123504 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:23:23.123535 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:23:23.123947 (+ 412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:23:23.123956 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:23:23.126115 (+ 2159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:23:23.126123 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:23:23.127766 (+ 1643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:23:23.127781 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:23:23.129464 (+ 1683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:23:23.129470 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:23:23.129672 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:23.129727 (+ 55us) client_negotiation.cc:769] Sending connection context
0929 20:23:23.129873 (+ 146us) client_negotiation.cc:240] Negotiation successful
0929 20:23:23.130075 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15119,"mutex_wait_us":6,"thread_start_us":13547,"threads_started":1}
I20260929 20:23:23.130570 25755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:23.105776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.66:40107 (local address 127.0.0.1:45766)
0929 20:23:23.106799 (+ 1023us) negotiation.cc:107] Waiting for socket to connect
0929 20:23:23.106879 (+ 80us) client_negotiation.cc:174] Beginning negotiation
0929 20:23:23.107091 (+ 212us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:23:23.123279 (+ 16188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:23:23.123283 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:23:23.123306 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:23:23.123857 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:23:23.123874 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:23:23.125736 (+ 1862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:23:23.125741 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:23:23.127958 (+ 2217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:23:23.127971 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:23:23.129763 (+ 1792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:23:23.129769 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:23:23.129933 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:23.129972 (+ 39us) client_negotiation.cc:769] Sending connection context
0929 20:23:23.130042 (+ 70us) client_negotiation.cc:240] Negotiation successful
0929 20:23:23.130188 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":687,"thread_start_us":525,"threads_started":1}
I20260929 20:23:23.131148 25758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:23.106645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47892 (local address 127.17.53.67:38515)
0929 20:23:23.122093 (+ 15448us) server_negotiation.cc:206] Beginning negotiation
0929 20:23:23.122102 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 20:23:23.122135 (+ 33us) server_negotiation.cc:407] Connection header received
0929 20:23:23.122252 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:23:23.122263 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:23:23.122594 (+ 331us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:23:23.122838 (+ 244us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:23:23.123529 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:23.125077 (+ 1548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:23.127993 (+ 2916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:23.129522 (+ 1529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:23.129670 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:23.129889 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0929 20:23:23.130614 (+ 725us) server_negotiation.cc:299] Negotiation successful
0929 20:23:23.130806 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15304,"thread_start_us":15189,"threads_started":1}
I20260929 20:23:23.131371 25759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:23.106031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.67:38515 (local address 127.0.0.1:47892)
0929 20:23:23.120383 (+ 14352us) negotiation.cc:107] Waiting for socket to connect
0929 20:23:23.120412 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0929 20:23:23.120606 (+ 194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:23:23.122838 (+ 2232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:23:23.122845 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:23:23.122895 (+ 50us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:23:23.123324 (+ 429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:23:23.123340 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:23:23.125241 (+ 1901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:23:23.125247 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:23:23.127792 (+ 2545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:23:23.127809 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:23:23.130028 (+ 2219us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:23:23.130036 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:23:23.130228 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:23.130255 (+ 27us) client_negotiation.cc:769] Sending connection context
0929 20:23:23.130335 (+ 80us) client_negotiation.cc:240] Negotiation successful
0929 20:23:23.130458 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14127,"mutex_wait_us":1,"thread_start_us":16006,"threads_started":1}
I20260929 20:23:23.131644 25756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:23.106145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45766 (local address 127.17.53.66:40107)
0929 20:23:23.122665 (+ 16520us) server_negotiation.cc:206] Beginning negotiation
0929 20:23:23.122674 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 20:23:23.122702 (+ 28us) server_negotiation.cc:407] Connection header received
0929 20:23:23.122801 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:23:23.122808 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:23:23.123086 (+ 278us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:23:23.123282 (+ 196us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:23:23.124058 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:23.125615 (+ 1557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:23.128129 (+ 2514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:23.129527 (+ 1398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:23.130804 (+ 1277us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:23.131038 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0929 20:23:23.131149 (+ 111us) server_negotiation.cc:299] Negotiation successful
0929 20:23:23.131327 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16358,"thread_start_us":16240,"threads_started":1}
I20260929 20:23:23.135136 25666 tablet_service.cc:1467] Processing CreateTablet for tablet b90979bafaa447b8ae9c91118f1a6d29 (DEFAULT_TABLE table=test-table [id=444dbc4202ab40978894419fef45b346]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:23:23.135151 25387 tablet_service.cc:1467] Processing CreateTablet for tablet b90979bafaa447b8ae9c91118f1a6d29 (DEFAULT_TABLE table=test-table [id=444dbc4202ab40978894419fef45b346]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:23:23.135151 25527 tablet_service.cc:1467] Processing CreateTablet for tablet b90979bafaa447b8ae9c91118f1a6d29 (DEFAULT_TABLE table=test-table [id=444dbc4202ab40978894419fef45b346]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:23:23.136456 25666 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b90979bafaa447b8ae9c91118f1a6d29. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:23:23.136528 25387 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b90979bafaa447b8ae9c91118f1a6d29. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:23:23.136520 25527 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b90979bafaa447b8ae9c91118f1a6d29. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:23:23.152304 25762 tablet_bootstrap.cc:492] T b90979bafaa447b8ae9c91118f1a6d29 P f3630496a5db48fda15d65cfcfcf175f: Bootstrap starting.
I20260929 20:23:23.154891 25761 tablet_bootstrap.cc:492] T b90979bafaa447b8ae9c91118f1a6d29 P f42427cf15e945249ac2061305a6f745: Bootstrap starting.
I20260929 20:23:23.156591 25763 tablet_bootstrap.cc:492] T b90979bafaa447b8ae9c91118f1a6d29 P 70e535e06c4b430c9ec90f98dbc4f69b: Bootstrap starting.
I20260929 20:23:23.156677 25762 tablet_bootstrap.cc:654] T b90979bafaa447b8ae9c91118f1a6d29 P f3630496a5db48fda15d65cfcfcf175f: Neither blocks nor log segments found. Creating new log.
I20260929 20:23:23.158186 25761 tablet_bootstrap.cc:654] T b90979bafaa447b8ae9c91118f1a6d29 P f42427cf15e945249ac2061305a6f745: Neither blocks nor log segments found. Creating new log.
I20260929 20:23:23.158407 25762 log.cc:824] T b90979bafaa447b8ae9c91118f1a6d29 P f3630496a5db48fda15d65cfcfcf175f: Log is configured to *not* fsync() on all Append() calls
I20260929 20:23:23.159458 25761 log.cc:824] T b90979bafaa447b8ae9c91118f1a6d29 P f42427cf15e945249ac2061305a6f745: Log is configured to *not* fsync() on all Append() calls
I20260929 20:23:23.160255 25763 tablet_bootstrap.cc:654] T b90979bafaa447b8ae9c91118f1a6d29 P 70e535e06c4b430c9ec90f98dbc4f69b: Neither blocks nor log segments found. Creating new log.
I20260929 20:23:23.161549 25763 log.cc:824] T b90979bafaa447b8ae9c91118f1a6d29 P 70e535e06c4b430c9ec90f98dbc4f69b: Log is configured to *not* fsync() on all Append() calls
I20260929 20:23:23.162626 25761 tablet_bootstrap.cc:492] T b90979bafaa447b8ae9c91118f1a6d29 P f42427cf15e945249ac2061305a6f745: No bootstrap required, opened a new log
I20260929 20:23:23.162828 25762 tablet_bootstrap.cc:492] T b90979bafaa447b8ae9c91118f1a6d29 P f3630496a5db48fda15d65cfcfcf175f: No bootstrap required, opened a new log
I20260929 20:23:23.162962 25761 ts_tablet_manager.cc:1397] T b90979bafaa447b8ae9c91118f1a6d29 P f42427cf15e945249ac2061305a6f745: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.003s
I20260929 20:23:23.163141 25762 ts_tablet_manager.cc:1397] T b90979bafaa447b8ae9c91118f1a6d29 P f3630496a5db48fda15d65cfcfcf175f: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.004s
I20260929 20:23:23.164459 25763 tablet_bootstrap.cc:492] T b90979bafaa447b8ae9c91118f1a6d29 P 70e535e06c4b430c9ec90f98dbc4f69b: No bootstrap required, opened a new log
I20260929 20:23:23.164736 25763 ts_tablet_manager.cc:1397] T b90979bafaa447b8ae9c91118f1a6d29 P 70e535e06c4b430c9ec90f98dbc4f69b: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260929 20:23:23.170105 25761 raft_consensus.cc:348] T b90979bafaa447b8ae9c91118f1a6d29 P f42427cf15e945249ac2061305a6f745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70e535e06c4b430c9ec90f98dbc4f69b" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 40107 } } peers { permanent_uuid: "f42427cf15e945249ac2061305a6f745" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 38515 } } peers { permanent_uuid: "f3630496a5db48fda15d65cfcfcf175f" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 41053 } }
I20260929 20:23:23.171027 25761 raft_consensus.cc:374] T b90979bafaa447b8ae9c91118f1a6d29 P f42427cf15e945249ac2061305a6f745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:23:23.171144 25761 raft_consensus.cc:729] T b90979bafaa447b8ae9c91118f1a6d29 P f42427cf15e945249ac2061305a6f745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f42427cf15e945249ac2061305a6f745, State: Initialized, Role: FOLLOWER
I20260929 20:23:23.171752 25761 consensus_queue.cc:260] T b90979bafaa447b8ae9c91118f1a6d29 P f42427cf15e945249ac2061305a6f745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70e535e06c4b430c9ec90f98dbc4f69b" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 40107 } } peers { permanent_uuid: "f42427cf15e945249ac2061305a6f745" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 38515 } } peers { permanent_uuid: "f3630496a5db48fda15d65cfcfcf175f" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 41053 } }
I20260929 20:23:23.173277 25761 ts_tablet_manager.cc:1428] T b90979bafaa447b8ae9c91118f1a6d29 P f42427cf15e945249ac2061305a6f745: Time spent starting tablet: real 0.010s user 0.003s sys 0.006s
I20260929 20:23:23.174576 25762 raft_consensus.cc:348] T b90979bafaa447b8ae9c91118f1a6d29 P f3630496a5db48fda15d65cfcfcf175f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70e535e06c4b430c9ec90f98dbc4f69b" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 40107 } } peers { permanent_uuid: "f42427cf15e945249ac2061305a6f745" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 38515 } } peers { permanent_uuid: "f3630496a5db48fda15d65cfcfcf175f" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 41053 } }
I20260929 20:23:23.175041 25762 raft_consensus.cc:374] T b90979bafaa447b8ae9c91118f1a6d29 P f3630496a5db48fda15d65cfcfcf175f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:23:23.175149 25762 raft_consensus.cc:729] T b90979bafaa447b8ae9c91118f1a6d29 P f3630496a5db48fda15d65cfcfcf175f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3630496a5db48fda15d65cfcfcf175f, State: Initialized, Role: FOLLOWER
I20260929 20:23:23.174953 25763 raft_consensus.cc:348] T b90979bafaa447b8ae9c91118f1a6d29 P 70e535e06c4b430c9ec90f98dbc4f69b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70e535e06c4b430c9ec90f98dbc4f69b" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 40107 } } peers { permanent_uuid: "f42427cf15e945249ac2061305a6f745" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 38515 } } peers { permanent_uuid: "f3630496a5db48fda15d65cfcfcf175f" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 41053 } }
I20260929 20:23:23.175375 25763 raft_consensus.cc:374] T b90979bafaa447b8ae9c91118f1a6d29 P 70e535e06c4b430c9ec90f98dbc4f69b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:23:23.175505 25763 raft_consensus.cc:729] T b90979bafaa447b8ae9c91118f1a6d29 P 70e535e06c4b430c9ec90f98dbc4f69b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 70e535e06c4b430c9ec90f98dbc4f69b, State: Initialized, Role: FOLLOWER
I20260929 20:23:23.175740 25762 consensus_queue.cc:260] T b90979bafaa447b8ae9c91118f1a6d29 P f3630496a5db48fda15d65cfcfcf175f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70e535e06c4b430c9ec90f98dbc4f69b" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 40107 } } peers { permanent_uuid: "f42427cf15e945249ac2061305a6f745" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 38515 } } peers { permanent_uuid: "f3630496a5db48fda15d65cfcfcf175f" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 41053 } }
I20260929 20:23:23.176082 25763 consensus_queue.cc:260] T b90979bafaa447b8ae9c91118f1a6d29 P 70e535e06c4b430c9ec90f98dbc4f69b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70e535e06c4b430c9ec90f98dbc4f69b" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 40107 } } peers { permanent_uuid: "f42427cf15e945249ac2061305a6f745" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 38515 } } peers { permanent_uuid: "f3630496a5db48fda15d65cfcfcf175f" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 41053 } }
I20260929 20:23:23.177489 25762 ts_tablet_manager.cc:1428] T b90979bafaa447b8ae9c91118f1a6d29 P f3630496a5db48fda15d65cfcfcf175f: Time spent starting tablet: real 0.014s user 0.010s sys 0.003s
I20260929 20:23:23.179164 25763 ts_tablet_manager.cc:1428] T b90979bafaa447b8ae9c91118f1a6d29 P 70e535e06c4b430c9ec90f98dbc4f69b: Time spent starting tablet: real 0.014s user 0.016s sys 0.000s
W20260929 20:23:23.190038 25736 tablet.cc:2369] T b90979bafaa447b8ae9c91118f1a6d29 P f42427cf15e945249ac2061305a6f745: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 20:23:23.363221 25458 tablet.cc:2369] T b90979bafaa447b8ae9c91118f1a6d29 P f3630496a5db48fda15d65cfcfcf175f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 20:23:23.408206 25767 raft_consensus.cc:482] T b90979bafaa447b8ae9c91118f1a6d29 P f42427cf15e945249ac2061305a6f745 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 20:23:23.408519 25767 raft_consensus.cc:504] T b90979bafaa447b8ae9c91118f1a6d29 P f42427cf15e945249ac2061305a6f745 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70e535e06c4b430c9ec90f98dbc4f69b" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 40107 } } peers { permanent_uuid: "f42427cf15e945249ac2061305a6f745" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 38515 } } peers { permanent_uuid: "f3630496a5db48fda15d65cfcfcf175f" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 41053 } }
I20260929 20:23:23.410104 25767 leader_election.cc:288] T b90979bafaa447b8ae9c91118f1a6d29 P f42427cf15e945249ac2061305a6f745 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 70e535e06c4b430c9ec90f98dbc4f69b (127.17.53.66:40107), f3630496a5db48fda15d65cfcfcf175f (127.17.53.65:41053)
I20260929 20:23:23.418921 25760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:23.411687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.53.67:39199 (local address 127.17.53.65:41053)
0929 20:23:23.411901 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0929 20:23:23.411907 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:23:23.412711 (+ 804us) server_negotiation.cc:407] Connection header received
0929 20:23:23.412994 (+ 283us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:23:23.413002 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:23:23.413081 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:23:23.413148 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:23:23.414010 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:23.415057 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:23.417182 (+ 2125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:23.417976 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:23.418166 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:23.418241 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0929 20:23:23.418525 (+ 284us) server_negotiation.cc:299] Negotiation successful
0929 20:23:23.418654 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260929 20:23:23.419003 25771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:23.411498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.65:41053 (local address 127.17.53.67:39199)
0929 20:23:23.412611 (+ 1113us) negotiation.cc:107] Waiting for socket to connect
0929 20:23:23.412627 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 20:23:23.412813 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:23:23.413360 (+ 547us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:23:23.413365 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:23:23.413390 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:23:23.413727 (+ 337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:23:23.413738 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:23:23.415211 (+ 1473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:23:23.415221 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:23:23.417033 (+ 1812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:23:23.417044 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:23:23.418183 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:23:23.418189 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:23:23.418358 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:23.418386 (+ 28us) client_negotiation.cc:769] Sending connection context
0929 20:23:23.418503 (+ 117us) client_negotiation.cc:240] Negotiation successful
0929 20:23:23.418638 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":983,"thread_start_us":810,"threads_started":1}
I20260929 20:23:23.419153 25756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:23.410516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.53.67:49515 (local address 127.17.53.66:40107)
0929 20:23:23.410702 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0929 20:23:23.410707 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:23:23.411433 (+ 726us) server_negotiation.cc:407] Connection header received
0929 20:23:23.411692 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:23:23.411698 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:23:23.411802 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:23:23.411936 (+ 134us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:23:23.413201 (+ 1265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:23.414307 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:23.416704 (+ 2397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:23.418021 (+ 1317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:23.418179 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:23.418249 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0929 20:23:23.418758 (+ 509us) server_negotiation.cc:299] Negotiation successful
0929 20:23:23.418919 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 20:23:23.419795 25770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:23.410735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.66:40107 (local address 127.17.53.67:49515)
0929 20:23:23.411319 (+ 584us) negotiation.cc:107] Waiting for socket to connect
0929 20:23:23.411335 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 20:23:23.411502 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:23:23.412571 (+ 1069us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:23:23.412575 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:23:23.412611 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:23:23.413025 (+ 414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:23:23.413036 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:23:23.414439 (+ 1403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:23:23.414447 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:23:23.416524 (+ 2077us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:23:23.416539 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:23:23.418401 (+ 1862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:23:23.418407 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:23:23.418584 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:23.418618 (+ 34us) client_negotiation.cc:769] Sending connection context
0929 20:23:23.419348 (+ 730us) client_negotiation.cc:240] Negotiation successful
0929 20:23:23.419487 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":470,"thread_start_us":310,"threads_started":1}
W20260929 20:23:23.420140 25598 tablet.cc:2369] T b90979bafaa447b8ae9c91118f1a6d29 P 70e535e06c4b430c9ec90f98dbc4f69b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 20:23:23.420398 25407 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b90979bafaa447b8ae9c91118f1a6d29" candidate_uuid: "f42427cf15e945249ac2061305a6f745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3630496a5db48fda15d65cfcfcf175f" is_pre_election: true
I20260929 20:23:23.420863 25551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b90979bafaa447b8ae9c91118f1a6d29" candidate_uuid: "f42427cf15e945249ac2061305a6f745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "70e535e06c4b430c9ec90f98dbc4f69b" is_pre_election: true
I20260929 20:23:23.421073 25407 raft_consensus.cc:2454] T b90979bafaa447b8ae9c91118f1a6d29 P f3630496a5db48fda15d65cfcfcf175f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f42427cf15e945249ac2061305a6f745 in term 0.
I20260929 20:23:23.421468 25551 raft_consensus.cc:2454] T b90979bafaa447b8ae9c91118f1a6d29 P 70e535e06c4b430c9ec90f98dbc4f69b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f42427cf15e945249ac2061305a6f745 in term 0.
I20260929 20:23:23.421792 25618 leader_election.cc:302] T b90979bafaa447b8ae9c91118f1a6d29 P f42427cf15e945249ac2061305a6f745 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f3630496a5db48fda15d65cfcfcf175f, f42427cf15e945249ac2061305a6f745; no voters:
I20260929 20:23:23.422263 25767 raft_consensus.cc:2781] T b90979bafaa447b8ae9c91118f1a6d29 P f42427cf15e945249ac2061305a6f745 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 20:23:23.422410 25767 raft_consensus.cc:482] T b90979bafaa447b8ae9c91118f1a6d29 P f42427cf15e945249ac2061305a6f745 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 20:23:23.422499 25767 raft_consensus.cc:3037] T b90979bafaa447b8ae9c91118f1a6d29 P f42427cf15e945249ac2061305a6f745 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:23:23.425261 25767 raft_consensus.cc:504] T b90979bafaa447b8ae9c91118f1a6d29 P f42427cf15e945249ac2061305a6f745 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70e535e06c4b430c9ec90f98dbc4f69b" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 40107 } } peers { permanent_uuid: "f42427cf15e945249ac2061305a6f745" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 38515 } } peers { permanent_uuid: "f3630496a5db48fda15d65cfcfcf175f" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 41053 } }
I20260929 20:23:23.425750 25767 leader_election.cc:288] T b90979bafaa447b8ae9c91118f1a6d29 P f42427cf15e945249ac2061305a6f745 [CANDIDATE]: Term 1 election: Requested vote from peers 70e535e06c4b430c9ec90f98dbc4f69b (127.17.53.66:40107), f3630496a5db48fda15d65cfcfcf175f (127.17.53.65:41053)
I20260929 20:23:23.426182 25551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b90979bafaa447b8ae9c91118f1a6d29" candidate_uuid: "f42427cf15e945249ac2061305a6f745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "70e535e06c4b430c9ec90f98dbc4f69b"
I20260929 20:23:23.426294 25407 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b90979bafaa447b8ae9c91118f1a6d29" candidate_uuid: "f42427cf15e945249ac2061305a6f745" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3630496a5db48fda15d65cfcfcf175f"
I20260929 20:23:23.426373 25551 raft_consensus.cc:3037] T b90979bafaa447b8ae9c91118f1a6d29 P 70e535e06c4b430c9ec90f98dbc4f69b [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:23:23.426514 25407 raft_consensus.cc:3037] T b90979bafaa447b8ae9c91118f1a6d29 P f3630496a5db48fda15d65cfcfcf175f [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:23:23.428578 25551 raft_consensus.cc:2454] T b90979bafaa447b8ae9c91118f1a6d29 P 70e535e06c4b430c9ec90f98dbc4f69b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f42427cf15e945249ac2061305a6f745 in term 1.
I20260929 20:23:23.428690 25407 raft_consensus.cc:2454] T b90979bafaa447b8ae9c91118f1a6d29 P f3630496a5db48fda15d65cfcfcf175f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f42427cf15e945249ac2061305a6f745 in term 1.
I20260929 20:23:23.428977 25618 leader_election.cc:302] T b90979bafaa447b8ae9c91118f1a6d29 P f42427cf15e945249ac2061305a6f745 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 70e535e06c4b430c9ec90f98dbc4f69b, f42427cf15e945249ac2061305a6f745; no voters:
I20260929 20:23:23.429250 25767 raft_consensus.cc:2781] T b90979bafaa447b8ae9c91118f1a6d29 P f42427cf15e945249ac2061305a6f745 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:23:23.429670 25767 raft_consensus.cc:686] T b90979bafaa447b8ae9c91118f1a6d29 P f42427cf15e945249ac2061305a6f745 [term 1 LEADER]: Becoming Leader. State: Replica: f42427cf15e945249ac2061305a6f745, State: Running, Role: LEADER
I20260929 20:23:23.430203 25767 consensus_queue.cc:237] T b90979bafaa447b8ae9c91118f1a6d29 P f42427cf15e945249ac2061305a6f745 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70e535e06c4b430c9ec90f98dbc4f69b" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 40107 } } peers { permanent_uuid: "f42427cf15e945249ac2061305a6f745" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 38515 } } peers { permanent_uuid: "f3630496a5db48fda15d65cfcfcf175f" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 41053 } }
I20260929 20:23:23.434726 25249 catalog_manager.cc:5495] T b90979bafaa447b8ae9c91118f1a6d29 P f42427cf15e945249ac2061305a6f745 reported cstate change: term changed from 0 to 1, leader changed from <none> to f42427cf15e945249ac2061305a6f745 (127.17.53.67). New cstate: current_term: 1 leader_uuid: "f42427cf15e945249ac2061305a6f745" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "70e535e06c4b430c9ec90f98dbc4f69b" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 40107 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f42427cf15e945249ac2061305a6f745" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 38515 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f3630496a5db48fda15d65cfcfcf175f" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 41053 } health_report { overall_health: UNKNOWN } } }
I20260929 20:23:23.496322 25758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:23.491051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47896 (local address 127.17.53.67:38515)
0929 20:23:23.491279 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0929 20:23:23.491285 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:23:23.491375 (+ 90us) server_negotiation.cc:407] Connection header received
0929 20:23:23.491661 (+ 286us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:23:23.491666 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:23:23.491749 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:23:23.491898 (+ 149us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 20:23:23.492260 (+ 362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:23.492967 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:23.494268 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:23.494666 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:23.494792 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:23.495232 (+ 440us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 20:23:23.495760 (+ 528us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 20:23:23.495911 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0929 20:23:23.495970 (+ 59us) server_negotiation.cc:299] Negotiation successful
0929 20:23:23.496058 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260929 20:23:23.503975 25551 raft_consensus.cc:1261] T b90979bafaa447b8ae9c91118f1a6d29 P 70e535e06c4b430c9ec90f98dbc4f69b [term 1 FOLLOWER]: Refusing update from remote peer f42427cf15e945249ac2061305a6f745: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:23:23.504050 25407 raft_consensus.cc:1261] T b90979bafaa447b8ae9c91118f1a6d29 P f3630496a5db48fda15d65cfcfcf175f [term 1 FOLLOWER]: Refusing update from remote peer f42427cf15e945249ac2061305a6f745: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:23:23.504925 25772 consensus_queue.cc:1035] T b90979bafaa447b8ae9c91118f1a6d29 P f42427cf15e945249ac2061305a6f745 [LEADER]: Connected to new peer: Peer: permanent_uuid: "70e535e06c4b430c9ec90f98dbc4f69b" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 40107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:23:23.505304 25767 consensus_queue.cc:1035] T b90979bafaa447b8ae9c91118f1a6d29 P f42427cf15e945249ac2061305a6f745 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f3630496a5db48fda15d65cfcfcf175f" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 41053 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:23:23.515064 25775 mvcc.cc:205] Tried to move back new op lower bound from 7334762100743385088 to 7334762100454535168. Current Snapshot: MvccSnapshot[applied={T|T < 7334762100743385088}]
I20260929 20:23:23.519490 25776 mvcc.cc:205] Tried to move back new op lower bound from 7334762100743385088 to 7334762100454535168. Current Snapshot: MvccSnapshot[applied={T|T < 7334762100743385088}]
I20260929 20:23:23.530824 25249 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57850:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 20:23:23.531121 25249 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57850:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 20:23:23.534734 25249 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d27fa57b2ddf4b438a0d3fd3a5271ac6: Sending DeleteTablet for 3 replicas of tablet b90979bafaa447b8ae9c91118f1a6d29
I20260929 20:23:23.535866 25527 tablet_service.cc:1514] Processing DeleteTablet for tablet b90979bafaa447b8ae9c91118f1a6d29 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:23:23 UTC) from {username='oryx', principal='oryx/127.17.53.126@KRBTEST.COM'} at 127.0.0.1:45766
I20260929 20:23:23.535878 25666 tablet_service.cc:1514] Processing DeleteTablet for tablet b90979bafaa447b8ae9c91118f1a6d29 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:23:23 UTC) from {username='oryx', principal='oryx/127.17.53.126@KRBTEST.COM'} at 127.0.0.1:47892
I20260929 20:23:23.536815 25785 tablet_replica.cc:330] T b90979bafaa447b8ae9c91118f1a6d29 P 70e535e06c4b430c9ec90f98dbc4f69b: stopping tablet replica
I20260929 20:23:23.537307 25785 raft_consensus.cc:2229] T b90979bafaa447b8ae9c91118f1a6d29 P 70e535e06c4b430c9ec90f98dbc4f69b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 20:23:23.537781 25785 raft_consensus.cc:2258] T b90979bafaa447b8ae9c91118f1a6d29 P 70e535e06c4b430c9ec90f98dbc4f69b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:23:23.539888 25387 tablet_service.cc:1514] Processing DeleteTablet for tablet b90979bafaa447b8ae9c91118f1a6d29 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:23:23 UTC) from {username='oryx', principal='oryx/127.17.53.126@KRBTEST.COM'} at 127.0.0.1:46666
I20260929 20:23:23.541731 25785 ts_tablet_manager.cc:1905] T b90979bafaa447b8ae9c91118f1a6d29 P 70e535e06c4b430c9ec90f98dbc4f69b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:23:23.542171 25249 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57850:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 20:23:23.543138 25249 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 20:23:23.543334 25786 tablet_replica.cc:330] T b90979bafaa447b8ae9c91118f1a6d29 P f42427cf15e945249ac2061305a6f745: stopping tablet replica
I20260929 20:23:23.543761 25786 raft_consensus.cc:2229] T b90979bafaa447b8ae9c91118f1a6d29 P f42427cf15e945249ac2061305a6f745 [term 1 LEADER]: Raft consensus shutting down.
I20260929 20:23:23.544363 25786 raft_consensus.cc:2258] T b90979bafaa447b8ae9c91118f1a6d29 P f42427cf15e945249ac2061305a6f745 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:23:23.550675 25787 tablet_replica.cc:330] T b90979bafaa447b8ae9c91118f1a6d29 P f3630496a5db48fda15d65cfcfcf175f: stopping tablet replica
I20260929 20:23:23.551043 25785 ts_tablet_manager.cc:1918] T b90979bafaa447b8ae9c91118f1a6d29 P 70e535e06c4b430c9ec90f98dbc4f69b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 20:23:23.551182 25787 raft_consensus.cc:2229] T b90979bafaa447b8ae9c91118f1a6d29 P f3630496a5db48fda15d65cfcfcf175f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 20:23:23.551285 25785 log.cc:1196] T b90979bafaa447b8ae9c91118f1a6d29 P 70e535e06c4b430c9ec90f98dbc4f69b: Deleting WAL directory at /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790713339161492-17621-0/minicluster-data/ts-1/wal/wals/b90979bafaa447b8ae9c91118f1a6d29
I20260929 20:23:23.551805 25787 raft_consensus.cc:2258] T b90979bafaa447b8ae9c91118f1a6d29 P f3630496a5db48fda15d65cfcfcf175f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:23:23.551851 25785 ts_tablet_manager.cc:1939] T b90979bafaa447b8ae9c91118f1a6d29 P 70e535e06c4b430c9ec90f98dbc4f69b: Deleting consensus metadata
I20260929 20:23:23.554142 25235 catalog_manager.cc:4841] TS 70e535e06c4b430c9ec90f98dbc4f69b (127.17.53.66:40107): tablet b90979bafaa447b8ae9c91118f1a6d29 (table test-table [id=444dbc4202ab40978894419fef45b346]) successfully deleted
I20260929 20:23:23.554929 25787 ts_tablet_manager.cc:1905] T b90979bafaa447b8ae9c91118f1a6d29 P f3630496a5db48fda15d65cfcfcf175f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:23:23.556484 25786 ts_tablet_manager.cc:1905] T b90979bafaa447b8ae9c91118f1a6d29 P f42427cf15e945249ac2061305a6f745: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:23:23.556908 25666 tablet_service.cc:1467] Processing CreateTablet for tablet 6588bacc31ea4864b73f3f626967a1b0 (DEFAULT_TABLE table=test-table [id=8bd8450d209d483a8b9121b70b9b7809]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:23:23.557080 25387 tablet_service.cc:1467] Processing CreateTablet for tablet 6588bacc31ea4864b73f3f626967a1b0 (DEFAULT_TABLE table=test-table [id=8bd8450d209d483a8b9121b70b9b7809]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:23:23.557348 25666 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6588bacc31ea4864b73f3f626967a1b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:23:23.557300 25527 tablet_service.cc:1467] Processing CreateTablet for tablet 6588bacc31ea4864b73f3f626967a1b0 (DEFAULT_TABLE table=test-table [id=8bd8450d209d483a8b9121b70b9b7809]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:23:23.557719 25527 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6588bacc31ea4864b73f3f626967a1b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:23:23.559080 25387 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6588bacc31ea4864b73f3f626967a1b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:23:23.564122 25761 tablet_bootstrap.cc:492] T 6588bacc31ea4864b73f3f626967a1b0 P f42427cf15e945249ac2061305a6f745: Bootstrap starting.
I20260929 20:23:23.564216 25786 ts_tablet_manager.cc:1918] T b90979bafaa447b8ae9c91118f1a6d29 P f42427cf15e945249ac2061305a6f745: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 20:23:23.564373 25786 log.cc:1196] T b90979bafaa447b8ae9c91118f1a6d29 P f42427cf15e945249ac2061305a6f745: Deleting WAL directory at /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790713339161492-17621-0/minicluster-data/ts-2/wal/wals/b90979bafaa447b8ae9c91118f1a6d29
I20260929 20:23:23.564731 25763 tablet_bootstrap.cc:492] T 6588bacc31ea4864b73f3f626967a1b0 P 70e535e06c4b430c9ec90f98dbc4f69b: Bootstrap starting.
I20260929 20:23:23.564985 25786 ts_tablet_manager.cc:1939] T b90979bafaa447b8ae9c91118f1a6d29 P f42427cf15e945249ac2061305a6f745: Deleting consensus metadata
I20260929 20:23:23.565749 25761 tablet_bootstrap.cc:654] T 6588bacc31ea4864b73f3f626967a1b0 P f42427cf15e945249ac2061305a6f745: Neither blocks nor log segments found. Creating new log.
I20260929 20:23:23.566283 25763 tablet_bootstrap.cc:654] T 6588bacc31ea4864b73f3f626967a1b0 P 70e535e06c4b430c9ec90f98dbc4f69b: Neither blocks nor log segments found. Creating new log.
I20260929 20:23:23.566745 25234 catalog_manager.cc:4841] TS f42427cf15e945249ac2061305a6f745 (127.17.53.67:38515): tablet b90979bafaa447b8ae9c91118f1a6d29 (table test-table [id=444dbc4202ab40978894419fef45b346]) successfully deleted
I20260929 20:23:23.573781 25787 ts_tablet_manager.cc:1918] T b90979bafaa447b8ae9c91118f1a6d29 P f3630496a5db48fda15d65cfcfcf175f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 20:23:23.573969 25787 log.cc:1196] T b90979bafaa447b8ae9c91118f1a6d29 P f3630496a5db48fda15d65cfcfcf175f: Deleting WAL directory at /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790713339161492-17621-0/minicluster-data/ts-0/wal/wals/b90979bafaa447b8ae9c91118f1a6d29
I20260929 20:23:23.574630 25787 ts_tablet_manager.cc:1939] T b90979bafaa447b8ae9c91118f1a6d29 P f3630496a5db48fda15d65cfcfcf175f: Deleting consensus metadata
I20260929 20:23:23.576822 25235 catalog_manager.cc:4841] TS f3630496a5db48fda15d65cfcfcf175f (127.17.53.65:41053): tablet b90979bafaa447b8ae9c91118f1a6d29 (table test-table [id=444dbc4202ab40978894419fef45b346]) successfully deleted
I20260929 20:23:23.576913 25762 tablet_bootstrap.cc:492] T 6588bacc31ea4864b73f3f626967a1b0 P f3630496a5db48fda15d65cfcfcf175f: Bootstrap starting.
I20260929 20:23:23.578655 25762 tablet_bootstrap.cc:654] T 6588bacc31ea4864b73f3f626967a1b0 P f3630496a5db48fda15d65cfcfcf175f: Neither blocks nor log segments found. Creating new log.
I20260929 20:23:23.579536 25761 tablet_bootstrap.cc:492] T 6588bacc31ea4864b73f3f626967a1b0 P f42427cf15e945249ac2061305a6f745: No bootstrap required, opened a new log
I20260929 20:23:23.579820 25761 ts_tablet_manager.cc:1397] T 6588bacc31ea4864b73f3f626967a1b0 P f42427cf15e945249ac2061305a6f745: Time spent bootstrapping tablet: real 0.016s user 0.000s sys 0.014s
I20260929 20:23:23.580108 25763 tablet_bootstrap.cc:492] T 6588bacc31ea4864b73f3f626967a1b0 P 70e535e06c4b430c9ec90f98dbc4f69b: No bootstrap required, opened a new log
I20260929 20:23:23.580303 25763 ts_tablet_manager.cc:1397] T 6588bacc31ea4864b73f3f626967a1b0 P 70e535e06c4b430c9ec90f98dbc4f69b: Time spent bootstrapping tablet: real 0.016s user 0.004s sys 0.007s
I20260929 20:23:23.581135 25763 raft_consensus.cc:348] T 6588bacc31ea4864b73f3f626967a1b0 P 70e535e06c4b430c9ec90f98dbc4f69b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3630496a5db48fda15d65cfcfcf175f" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 41053 } } peers { permanent_uuid: "70e535e06c4b430c9ec90f98dbc4f69b" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 40107 } } peers { permanent_uuid: "f42427cf15e945249ac2061305a6f745" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 38515 } }
I20260929 20:23:23.581092 25761 raft_consensus.cc:348] T 6588bacc31ea4864b73f3f626967a1b0 P f42427cf15e945249ac2061305a6f745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3630496a5db48fda15d65cfcfcf175f" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 41053 } } peers { permanent_uuid: "70e535e06c4b430c9ec90f98dbc4f69b" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 40107 } } peers { permanent_uuid: "f42427cf15e945249ac2061305a6f745" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 38515 } }
I20260929 20:23:23.581403 25763 raft_consensus.cc:374] T 6588bacc31ea4864b73f3f626967a1b0 P 70e535e06c4b430c9ec90f98dbc4f69b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:23:23.581502 25761 raft_consensus.cc:374] T 6588bacc31ea4864b73f3f626967a1b0 P f42427cf15e945249ac2061305a6f745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:23:23.581527 25763 raft_consensus.cc:729] T 6588bacc31ea4864b73f3f626967a1b0 P 70e535e06c4b430c9ec90f98dbc4f69b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 70e535e06c4b430c9ec90f98dbc4f69b, State: Initialized, Role: FOLLOWER
I20260929 20:23:23.581724 25761 raft_consensus.cc:729] T 6588bacc31ea4864b73f3f626967a1b0 P f42427cf15e945249ac2061305a6f745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f42427cf15e945249ac2061305a6f745, State: Initialized, Role: FOLLOWER
I20260929 20:23:23.581822 25763 consensus_queue.cc:260] T 6588bacc31ea4864b73f3f626967a1b0 P 70e535e06c4b430c9ec90f98dbc4f69b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3630496a5db48fda15d65cfcfcf175f" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 41053 } } peers { permanent_uuid: "70e535e06c4b430c9ec90f98dbc4f69b" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 40107 } } peers { permanent_uuid: "f42427cf15e945249ac2061305a6f745" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 38515 } }
I20260929 20:23:23.582733 25761 consensus_queue.cc:260] T 6588bacc31ea4864b73f3f626967a1b0 P f42427cf15e945249ac2061305a6f745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3630496a5db48fda15d65cfcfcf175f" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 41053 } } peers { permanent_uuid: "70e535e06c4b430c9ec90f98dbc4f69b" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 40107 } } peers { permanent_uuid: "f42427cf15e945249ac2061305a6f745" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 38515 } }
I20260929 20:23:23.584784 25761 ts_tablet_manager.cc:1428] T 6588bacc31ea4864b73f3f626967a1b0 P f42427cf15e945249ac2061305a6f745: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20260929 20:23:23.586920 25763 ts_tablet_manager.cc:1428] T 6588bacc31ea4864b73f3f626967a1b0 P 70e535e06c4b430c9ec90f98dbc4f69b: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260929 20:23:23.596210 25762 tablet_bootstrap.cc:492] T 6588bacc31ea4864b73f3f626967a1b0 P f3630496a5db48fda15d65cfcfcf175f: No bootstrap required, opened a new log
I20260929 20:23:23.596392 25762 ts_tablet_manager.cc:1397] T 6588bacc31ea4864b73f3f626967a1b0 P f3630496a5db48fda15d65cfcfcf175f: Time spent bootstrapping tablet: real 0.020s user 0.007s sys 0.011s
I20260929 20:23:23.597066 25762 raft_consensus.cc:348] T 6588bacc31ea4864b73f3f626967a1b0 P f3630496a5db48fda15d65cfcfcf175f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3630496a5db48fda15d65cfcfcf175f" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 41053 } } peers { permanent_uuid: "70e535e06c4b430c9ec90f98dbc4f69b" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 40107 } } peers { permanent_uuid: "f42427cf15e945249ac2061305a6f745" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 38515 } }
I20260929 20:23:23.597255 25762 raft_consensus.cc:374] T 6588bacc31ea4864b73f3f626967a1b0 P f3630496a5db48fda15d65cfcfcf175f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:23:23.597368 25762 raft_consensus.cc:729] T 6588bacc31ea4864b73f3f626967a1b0 P f3630496a5db48fda15d65cfcfcf175f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3630496a5db48fda15d65cfcfcf175f, State: Initialized, Role: FOLLOWER
I20260929 20:23:23.597646 25762 consensus_queue.cc:260] T 6588bacc31ea4864b73f3f626967a1b0 P f3630496a5db48fda15d65cfcfcf175f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3630496a5db48fda15d65cfcfcf175f" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 41053 } } peers { permanent_uuid: "70e535e06c4b430c9ec90f98dbc4f69b" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 40107 } } peers { permanent_uuid: "f42427cf15e945249ac2061305a6f745" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 38515 } }
I20260929 20:23:23.598198 25762 ts_tablet_manager.cc:1428] T 6588bacc31ea4864b73f3f626967a1b0 P f3630496a5db48fda15d65cfcfcf175f: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260929 20:23:23.621025 25768 raft_consensus.cc:482] T 6588bacc31ea4864b73f3f626967a1b0 P f3630496a5db48fda15d65cfcfcf175f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 20:23:23.621275 25768 raft_consensus.cc:504] T 6588bacc31ea4864b73f3f626967a1b0 P f3630496a5db48fda15d65cfcfcf175f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3630496a5db48fda15d65cfcfcf175f" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 41053 } } peers { permanent_uuid: "70e535e06c4b430c9ec90f98dbc4f69b" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 40107 } } peers { permanent_uuid: "f42427cf15e945249ac2061305a6f745" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 38515 } }
I20260929 20:23:23.623219 25768 leader_election.cc:288] T 6588bacc31ea4864b73f3f626967a1b0 P f3630496a5db48fda15d65cfcfcf175f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 70e535e06c4b430c9ec90f98dbc4f69b (127.17.53.66:40107), f42427cf15e945249ac2061305a6f745 (127.17.53.67:38515)
I20260929 20:23:23.632404 25756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:23.623985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.53.65:37493 (local address 127.17.53.66:40107)
0929 20:23:23.624232 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0929 20:23:23.624239 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:23:23.625166 (+ 927us) server_negotiation.cc:407] Connection header received
0929 20:23:23.625476 (+ 310us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:23:23.625484 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:23:23.625624 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:23:23.625729 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:23:23.626479 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:23.627634 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:23.630123 (+ 2489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:23.631253 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:23.631445 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:23.631526 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0929 20:23:23.632016 (+ 490us) server_negotiation.cc:299] Negotiation successful
0929 20:23:23.632140 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260929 20:23:23.632555 25791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:23.623708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.66:40107 (local address 127.17.53.65:37493)
0929 20:23:23.625032 (+ 1324us) negotiation.cc:107] Waiting for socket to connect
0929 20:23:23.625058 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0929 20:23:23.625283 (+ 225us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:23:23.625778 (+ 495us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:23:23.625783 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:23:23.625811 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:23:23.626284 (+ 473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:23:23.626299 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:23:23.627766 (+ 1467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:23:23.627773 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:23:23.629922 (+ 2149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:23:23.629938 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:23:23.631492 (+ 1554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:23:23.631499 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:23:23.631775 (+ 276us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:23.631828 (+ 53us) client_negotiation.cc:769] Sending connection context
0929 20:23:23.631938 (+ 110us) client_negotiation.cc:240] Negotiation successful
0929 20:23:23.632076 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1156,"mutex_wait_us":27,"thread_start_us":1196,"threads_started":1}
I20260929 20:23:23.632741 25758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:23.624641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.53.65:35327 (local address 127.17.53.67:38515)
0929 20:23:23.624820 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0929 20:23:23.624825 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:23:23.625359 (+ 534us) server_negotiation.cc:407] Connection header received
0929 20:23:23.625549 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:23:23.625556 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:23:23.625672 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:23:23.625765 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:23:23.626585 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:23.627370 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:23.630097 (+ 2727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:23.631478 (+ 1381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:23.631651 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:23.631720 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0929 20:23:23.632267 (+ 547us) server_negotiation.cc:299] Negotiation successful
0929 20:23:23.632409 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 20:23:23.633316 25551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6588bacc31ea4864b73f3f626967a1b0" candidate_uuid: "f3630496a5db48fda15d65cfcfcf175f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "70e535e06c4b430c9ec90f98dbc4f69b" is_pre_election: true
I20260929 20:23:23.633579 25792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:23.624140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.67:38515 (local address 127.17.53.65:35327)
0929 20:23:23.625263 (+ 1123us) negotiation.cc:107] Waiting for socket to connect
0929 20:23:23.625278 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 20:23:23.625386 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:23:23.625974 (+ 588us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:23:23.625979 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:23:23.625998 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:23:23.626459 (+ 461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:23:23.626471 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:23:23.627501 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:23:23.627508 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:23:23.629922 (+ 2414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:23:23.629938 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:23:23.631878 (+ 1940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:23:23.631885 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:23:23.632078 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:23.632112 (+ 34us) client_negotiation.cc:769] Sending connection context
0929 20:23:23.633053 (+ 941us) client_negotiation.cc:240] Negotiation successful
0929 20:23:23.633200 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1009,"mutex_wait_us":114,"thread_start_us":930,"threads_started":1}
I20260929 20:23:23.633565 25551 raft_consensus.cc:2454] T 6588bacc31ea4864b73f3f626967a1b0 P 70e535e06c4b430c9ec90f98dbc4f69b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f3630496a5db48fda15d65cfcfcf175f in term 0.
I20260929 20:23:23.634126 25338 leader_election.cc:302] T 6588bacc31ea4864b73f3f626967a1b0 P f3630496a5db48fda15d65cfcfcf175f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 70e535e06c4b430c9ec90f98dbc4f69b, f3630496a5db48fda15d65cfcfcf175f; no voters:
I20260929 20:23:23.634586 25768 raft_consensus.cc:2781] T 6588bacc31ea4864b73f3f626967a1b0 P f3630496a5db48fda15d65cfcfcf175f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 20:23:23.634470 25686 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6588bacc31ea4864b73f3f626967a1b0" candidate_uuid: "f3630496a5db48fda15d65cfcfcf175f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f42427cf15e945249ac2061305a6f745" is_pre_election: true
I20260929 20:23:23.634712 25768 raft_consensus.cc:482] T 6588bacc31ea4864b73f3f626967a1b0 P f3630496a5db48fda15d65cfcfcf175f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 20:23:23.634809 25768 raft_consensus.cc:3037] T 6588bacc31ea4864b73f3f626967a1b0 P f3630496a5db48fda15d65cfcfcf175f [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:23:23.634975 25686 raft_consensus.cc:2454] T 6588bacc31ea4864b73f3f626967a1b0 P f42427cf15e945249ac2061305a6f745 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f3630496a5db48fda15d65cfcfcf175f in term 0.
I20260929 20:23:23.638041 25768 raft_consensus.cc:504] T 6588bacc31ea4864b73f3f626967a1b0 P f3630496a5db48fda15d65cfcfcf175f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3630496a5db48fda15d65cfcfcf175f" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 41053 } } peers { permanent_uuid: "70e535e06c4b430c9ec90f98dbc4f69b" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 40107 } } peers { permanent_uuid: "f42427cf15e945249ac2061305a6f745" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 38515 } }
I20260929 20:23:23.638972 25768 leader_election.cc:288] T 6588bacc31ea4864b73f3f626967a1b0 P f3630496a5db48fda15d65cfcfcf175f [CANDIDATE]: Term 1 election: Requested vote from peers 70e535e06c4b430c9ec90f98dbc4f69b (127.17.53.66:40107), f42427cf15e945249ac2061305a6f745 (127.17.53.67:38515)
I20260929 20:23:23.639571 25686 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6588bacc31ea4864b73f3f626967a1b0" candidate_uuid: "f3630496a5db48fda15d65cfcfcf175f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f42427cf15e945249ac2061305a6f745"
I20260929 20:23:23.639600 25551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6588bacc31ea4864b73f3f626967a1b0" candidate_uuid: "f3630496a5db48fda15d65cfcfcf175f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "70e535e06c4b430c9ec90f98dbc4f69b"
I20260929 20:23:23.639802 25686 raft_consensus.cc:3037] T 6588bacc31ea4864b73f3f626967a1b0 P f42427cf15e945249ac2061305a6f745 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:23:23.639806 25551 raft_consensus.cc:3037] T 6588bacc31ea4864b73f3f626967a1b0 P 70e535e06c4b430c9ec90f98dbc4f69b [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:23:23.642500 25686 raft_consensus.cc:2454] T 6588bacc31ea4864b73f3f626967a1b0 P f42427cf15e945249ac2061305a6f745 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f3630496a5db48fda15d65cfcfcf175f in term 1.
I20260929 20:23:23.642787 25551 raft_consensus.cc:2454] T 6588bacc31ea4864b73f3f626967a1b0 P 70e535e06c4b430c9ec90f98dbc4f69b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f3630496a5db48fda15d65cfcfcf175f in term 1.
I20260929 20:23:23.642966 25337 leader_election.cc:302] T 6588bacc31ea4864b73f3f626967a1b0 P f3630496a5db48fda15d65cfcfcf175f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f3630496a5db48fda15d65cfcfcf175f, f42427cf15e945249ac2061305a6f745; no voters:
I20260929 20:23:23.643513 25768 raft_consensus.cc:2781] T 6588bacc31ea4864b73f3f626967a1b0 P f3630496a5db48fda15d65cfcfcf175f [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:23:23.643980 25768 raft_consensus.cc:686] T 6588bacc31ea4864b73f3f626967a1b0 P f3630496a5db48fda15d65cfcfcf175f [term 1 LEADER]: Becoming Leader. State: Replica: f3630496a5db48fda15d65cfcfcf175f, State: Running, Role: LEADER
I20260929 20:23:23.644410 25768 consensus_queue.cc:237] T 6588bacc31ea4864b73f3f626967a1b0 P f3630496a5db48fda15d65cfcfcf175f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3630496a5db48fda15d65cfcfcf175f" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 41053 } } peers { permanent_uuid: "70e535e06c4b430c9ec90f98dbc4f69b" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 40107 } } peers { permanent_uuid: "f42427cf15e945249ac2061305a6f745" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 38515 } }
I20260929 20:23:23.648190 25250 catalog_manager.cc:5495] T 6588bacc31ea4864b73f3f626967a1b0 P f3630496a5db48fda15d65cfcfcf175f reported cstate change: term changed from 0 to 1, leader changed from <none> to f3630496a5db48fda15d65cfcfcf175f (127.17.53.65). New cstate: current_term: 1 leader_uuid: "f3630496a5db48fda15d65cfcfcf175f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3630496a5db48fda15d65cfcfcf175f" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 41053 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "70e535e06c4b430c9ec90f98dbc4f69b" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 40107 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f42427cf15e945249ac2061305a6f745" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 38515 } health_report { overall_health: UNKNOWN } } }
Sep 29 20:23:23 dist-test-slave-zbnc krb5kdc[25205](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713400, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.53.126@KRBTEST.COM for oryx/127.17.53.126@KRBTEST.COM
I20260929 20:23:23.708222 25800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:23.693319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:36941 (local address 127.0.0.1:57854)
0929 20:23:23.694413 (+ 1094us) negotiation.cc:107] Waiting for socket to connect
0929 20:23:23.694434 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 20:23:23.694558 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:23:23.694960 (+ 402us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:23:23.694963 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:23:23.695590 (+ 627us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:23:23.695803 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:23:23.695812 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:23:23.697323 (+ 1511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:23:23.697326 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:23:23.698258 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:23:23.698267 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:23:23.699241 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:23:23.699245 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:23:23.699371 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:23.700043 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:23:23.700059 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:23:23.702203 (+ 2144us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:23:23.705903 (+ 3700us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:23:23.705912 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:23:23.705917 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:23:23.706318 (+ 401us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:23:23.706698 (+ 380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:23:23.706701 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:23:23.706703 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:23:23.706826 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:23:23.707445 (+ 619us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:23:23.707451 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:23:23.707600 (+ 149us) client_negotiation.cc:769] Sending connection context
0929 20:23:23.707759 (+ 159us) client_negotiation.cc:240] Negotiation successful
0929 20:23:23.707957 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":963,"thread_start_us":826,"threads_started":1}
I20260929 20:23:23.708559 25801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:23.693351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57854 (local address 127.17.53.126:36941)
0929 20:23:23.694274 (+ 923us) server_negotiation.cc:206] Beginning negotiation
0929 20:23:23.694287 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 20:23:23.694527 (+ 240us) server_negotiation.cc:407] Connection header received
0929 20:23:23.694673 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:23:23.694679 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:23:23.694755 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:23:23.694898 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:23:23.695957 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:23.697205 (+ 1248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:23.698397 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:23.699084 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:23.699211 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:23.702405 (+ 3194us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:23:23.702412 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:23:23.702418 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:23:23.702453 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:23:23.705774 (+ 3321us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:23:23.706426 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:23:23.706431 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:23:23.706435 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:23:23.706579 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:23:23.706973 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:23:23.706979 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:23:23.706982 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:23:23.707324 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:23:23.707457 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0929 20:23:23.707889 (+ 432us) server_negotiation.cc:299] Negotiation successful
0929 20:23:23.708106 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":757,"thread_start_us":504,"threads_started":1}
I20260929 20:23:23.713442 25250 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.17.53.126@KRBTEST.COM'} at 127.0.0.1:57854:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260929 20:23:23.714699 25250 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 20:23:23.721652 25387 tablet_service.cc:1467] Processing CreateTablet for tablet 1e69a3414cee4903a74610495a89ce1f (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=b577ec4fe93041d5b3c5fc511241e0a1]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 20:23:23.721676 25527 tablet_service.cc:1467] Processing CreateTablet for tablet 1e69a3414cee4903a74610495a89ce1f (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=b577ec4fe93041d5b3c5fc511241e0a1]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 20:23:23.721760 25666 tablet_service.cc:1467] Processing CreateTablet for tablet 1e69a3414cee4903a74610495a89ce1f (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=b577ec4fe93041d5b3c5fc511241e0a1]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260929 20:23:23.722240 25527 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1e69a3414cee4903a74610495a89ce1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:23:23.722257 25387 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1e69a3414cee4903a74610495a89ce1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:23:23.722268 25666 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1e69a3414cee4903a74610495a89ce1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:23:23.729604 25763 tablet_bootstrap.cc:492] T 1e69a3414cee4903a74610495a89ce1f P 70e535e06c4b430c9ec90f98dbc4f69b: Bootstrap starting.
I20260929 20:23:23.729707 25761 tablet_bootstrap.cc:492] T 1e69a3414cee4903a74610495a89ce1f P f42427cf15e945249ac2061305a6f745: Bootstrap starting.
I20260929 20:23:23.730763 25762 tablet_bootstrap.cc:492] T 1e69a3414cee4903a74610495a89ce1f P f3630496a5db48fda15d65cfcfcf175f: Bootstrap starting.
I20260929 20:23:23.731331 25763 tablet_bootstrap.cc:654] T 1e69a3414cee4903a74610495a89ce1f P 70e535e06c4b430c9ec90f98dbc4f69b: Neither blocks nor log segments found. Creating new log.
I20260929 20:23:23.731478 25761 tablet_bootstrap.cc:654] T 1e69a3414cee4903a74610495a89ce1f P f42427cf15e945249ac2061305a6f745: Neither blocks nor log segments found. Creating new log.
I20260929 20:23:23.732607 25762 tablet_bootstrap.cc:654] T 1e69a3414cee4903a74610495a89ce1f P f3630496a5db48fda15d65cfcfcf175f: Neither blocks nor log segments found. Creating new log.
I20260929 20:23:23.740643 25762 tablet_bootstrap.cc:492] T 1e69a3414cee4903a74610495a89ce1f P f3630496a5db48fda15d65cfcfcf175f: No bootstrap required, opened a new log
I20260929 20:23:23.741065 25763 tablet_bootstrap.cc:492] T 1e69a3414cee4903a74610495a89ce1f P 70e535e06c4b430c9ec90f98dbc4f69b: No bootstrap required, opened a new log
I20260929 20:23:23.741087 25762 ts_tablet_manager.cc:1397] T 1e69a3414cee4903a74610495a89ce1f P f3630496a5db48fda15d65cfcfcf175f: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.005s
I20260929 20:23:23.741321 25763 ts_tablet_manager.cc:1397] T 1e69a3414cee4903a74610495a89ce1f P 70e535e06c4b430c9ec90f98dbc4f69b: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.003s
I20260929 20:23:23.742146 25763 raft_consensus.cc:348] T 1e69a3414cee4903a74610495a89ce1f P 70e535e06c4b430c9ec90f98dbc4f69b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3630496a5db48fda15d65cfcfcf175f" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 41053 } } peers { permanent_uuid: "70e535e06c4b430c9ec90f98dbc4f69b" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 40107 } } peers { permanent_uuid: "f42427cf15e945249ac2061305a6f745" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 38515 } }
I20260929 20:23:23.742404 25763 raft_consensus.cc:374] T 1e69a3414cee4903a74610495a89ce1f P 70e535e06c4b430c9ec90f98dbc4f69b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:23:23.742305 25762 raft_consensus.cc:348] T 1e69a3414cee4903a74610495a89ce1f P f3630496a5db48fda15d65cfcfcf175f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3630496a5db48fda15d65cfcfcf175f" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 41053 } } peers { permanent_uuid: "70e535e06c4b430c9ec90f98dbc4f69b" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 40107 } } peers { permanent_uuid: "f42427cf15e945249ac2061305a6f745" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 38515 } }
I20260929 20:23:23.742487 25763 raft_consensus.cc:729] T 1e69a3414cee4903a74610495a89ce1f P 70e535e06c4b430c9ec90f98dbc4f69b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 70e535e06c4b430c9ec90f98dbc4f69b, State: Initialized, Role: FOLLOWER
I20260929 20:23:23.742561 25762 raft_consensus.cc:374] T 1e69a3414cee4903a74610495a89ce1f P f3630496a5db48fda15d65cfcfcf175f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:23:23.742645 25762 raft_consensus.cc:729] T 1e69a3414cee4903a74610495a89ce1f P f3630496a5db48fda15d65cfcfcf175f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3630496a5db48fda15d65cfcfcf175f, State: Initialized, Role: FOLLOWER
I20260929 20:23:23.742837 25763 consensus_queue.cc:260] T 1e69a3414cee4903a74610495a89ce1f P 70e535e06c4b430c9ec90f98dbc4f69b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3630496a5db48fda15d65cfcfcf175f" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 41053 } } peers { permanent_uuid: "70e535e06c4b430c9ec90f98dbc4f69b" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 40107 } } peers { permanent_uuid: "f42427cf15e945249ac2061305a6f745" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 38515 } }
I20260929 20:23:23.742909 25762 consensus_queue.cc:260] T 1e69a3414cee4903a74610495a89ce1f P f3630496a5db48fda15d65cfcfcf175f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3630496a5db48fda15d65cfcfcf175f" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 41053 } } peers { permanent_uuid: "70e535e06c4b430c9ec90f98dbc4f69b" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 40107 } } peers { permanent_uuid: "f42427cf15e945249ac2061305a6f745" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 38515 } }
I20260929 20:23:23.743556 25763 ts_tablet_manager.cc:1428] T 1e69a3414cee4903a74610495a89ce1f P 70e535e06c4b430c9ec90f98dbc4f69b: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260929 20:23:23.743410 25768 tablet_replica.cc:439] T 1e69a3414cee4903a74610495a89ce1f P f3630496a5db48fda15d65cfcfcf175f: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3630496a5db48fda15d65cfcfcf175f" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 41053 } } peers { permanent_uuid: "70e535e06c4b430c9ec90f98dbc4f69b" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 40107 } } peers { permanent_uuid: "f42427cf15e945249ac2061305a6f745" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 38515 } } }
I20260929 20:23:23.743706 25768 tablet_replica.cc:442] T 1e69a3414cee4903a74610495a89ce1f P f3630496a5db48fda15d65cfcfcf175f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 20:23:23.743440 25769 tablet_replica.cc:439] T 1e69a3414cee4903a74610495a89ce1f P 70e535e06c4b430c9ec90f98dbc4f69b: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3630496a5db48fda15d65cfcfcf175f" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 41053 } } peers { permanent_uuid: "70e535e06c4b430c9ec90f98dbc4f69b" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 40107 } } peers { permanent_uuid: "f42427cf15e945249ac2061305a6f745" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 38515 } } }
I20260929 20:23:23.743817 25769 tablet_replica.cc:442] T 1e69a3414cee4903a74610495a89ce1f P 70e535e06c4b430c9ec90f98dbc4f69b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 20:23:23.745736 25762 ts_tablet_manager.cc:1428] T 1e69a3414cee4903a74610495a89ce1f P f3630496a5db48fda15d65cfcfcf175f: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20260929 20:23:23.747197 25761 tablet_bootstrap.cc:492] T 1e69a3414cee4903a74610495a89ce1f P f42427cf15e945249ac2061305a6f745: No bootstrap required, opened a new log
I20260929 20:23:23.747385 25761 ts_tablet_manager.cc:1397] T 1e69a3414cee4903a74610495a89ce1f P f42427cf15e945249ac2061305a6f745: Time spent bootstrapping tablet: real 0.018s user 0.005s sys 0.005s
I20260929 20:23:23.748262 25761 raft_consensus.cc:348] T 1e69a3414cee4903a74610495a89ce1f P f42427cf15e945249ac2061305a6f745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3630496a5db48fda15d65cfcfcf175f" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 41053 } } peers { permanent_uuid: "70e535e06c4b430c9ec90f98dbc4f69b" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 40107 } } peers { permanent_uuid: "f42427cf15e945249ac2061305a6f745" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 38515 } }
I20260929 20:23:23.748539 25761 raft_consensus.cc:374] T 1e69a3414cee4903a74610495a89ce1f P f42427cf15e945249ac2061305a6f745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:23:23.748633 25761 raft_consensus.cc:729] T 1e69a3414cee4903a74610495a89ce1f P f42427cf15e945249ac2061305a6f745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f42427cf15e945249ac2061305a6f745, State: Initialized, Role: FOLLOWER
I20260929 20:23:23.748975 25761 consensus_queue.cc:260] T 1e69a3414cee4903a74610495a89ce1f P f42427cf15e945249ac2061305a6f745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3630496a5db48fda15d65cfcfcf175f" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 41053 } } peers { permanent_uuid: "70e535e06c4b430c9ec90f98dbc4f69b" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 40107 } } peers { permanent_uuid: "f42427cf15e945249ac2061305a6f745" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 38515 } }
I20260929 20:23:23.749691 25761 ts_tablet_manager.cc:1428] T 1e69a3414cee4903a74610495a89ce1f P f42427cf15e945249ac2061305a6f745: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260929 20:23:23.749477 25772 tablet_replica.cc:439] T 1e69a3414cee4903a74610495a89ce1f P f42427cf15e945249ac2061305a6f745: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3630496a5db48fda15d65cfcfcf175f" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 41053 } } peers { permanent_uuid: "70e535e06c4b430c9ec90f98dbc4f69b" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 40107 } } peers { permanent_uuid: "f42427cf15e945249ac2061305a6f745" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 38515 } } }
I20260929 20:23:23.750311 25772 tablet_replica.cc:442] T 1e69a3414cee4903a74610495a89ce1f P f42427cf15e945249ac2061305a6f745: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 20:23:23.844249 25769 raft_consensus.cc:482] T 1e69a3414cee4903a74610495a89ce1f P 70e535e06c4b430c9ec90f98dbc4f69b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 20:23:23.844457 25769 raft_consensus.cc:504] T 1e69a3414cee4903a74610495a89ce1f P 70e535e06c4b430c9ec90f98dbc4f69b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3630496a5db48fda15d65cfcfcf175f" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 41053 } } peers { permanent_uuid: "70e535e06c4b430c9ec90f98dbc4f69b" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 40107 } } peers { permanent_uuid: "f42427cf15e945249ac2061305a6f745" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 38515 } }
I20260929 20:23:23.846274 25769 leader_election.cc:288] T 1e69a3414cee4903a74610495a89ce1f P 70e535e06c4b430c9ec90f98dbc4f69b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f3630496a5db48fda15d65cfcfcf175f (127.17.53.65:41053), f42427cf15e945249ac2061305a6f745 (127.17.53.67:38515)
I20260929 20:23:23.855012 25760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:23.846985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.53.66:42339 (local address 127.17.53.65:41053)
0929 20:23:23.847222 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0929 20:23:23.847231 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 20:23:23.848122 (+ 891us) server_negotiation.cc:407] Connection header received
0929 20:23:23.848271 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:23:23.848277 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:23:23.848357 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:23:23.848457 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:23:23.849240 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:23.849973 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:23.852836 (+ 2863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:23.854089 (+ 1253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:23.854284 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:23.854366 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0929 20:23:23.854578 (+ 212us) server_negotiation.cc:299] Negotiation successful
0929 20:23:23.854713 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260929 20:23:23.855104 25805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:23.846675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.65:41053 (local address 127.17.53.66:42339)
0929 20:23:23.848017 (+ 1342us) negotiation.cc:107] Waiting for socket to connect
0929 20:23:23.848032 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 20:23:23.848142 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:23:23.848605 (+ 463us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:23:23.848609 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:23:23.848634 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:23:23.849085 (+ 451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:23:23.849094 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:23:23.850103 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:23:23.850110 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:23:23.852660 (+ 2550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:23:23.852675 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:23:23.854297 (+ 1622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:23:23.854302 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:23:23.854410 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:23.854434 (+ 24us) client_negotiation.cc:769] Sending connection context
0929 20:23:23.854537 (+ 103us) client_negotiation.cc:240] Negotiation successful
0929 20:23:23.854673 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1240,"mutex_wait_us":107,"thread_start_us":718,"threads_started":1}
I20260929 20:23:23.855856 25407 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e69a3414cee4903a74610495a89ce1f" candidate_uuid: "70e535e06c4b430c9ec90f98dbc4f69b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3630496a5db48fda15d65cfcfcf175f" is_pre_election: true
I20260929 20:23:23.856105 25407 raft_consensus.cc:2454] T 1e69a3414cee4903a74610495a89ce1f P f3630496a5db48fda15d65cfcfcf175f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 70e535e06c4b430c9ec90f98dbc4f69b in term 0.
I20260929 20:23:23.856585 25478 leader_election.cc:302] T 1e69a3414cee4903a74610495a89ce1f P 70e535e06c4b430c9ec90f98dbc4f69b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 70e535e06c4b430c9ec90f98dbc4f69b, f3630496a5db48fda15d65cfcfcf175f; no voters:
I20260929 20:23:23.857020 25769 raft_consensus.cc:2781] T 1e69a3414cee4903a74610495a89ce1f P 70e535e06c4b430c9ec90f98dbc4f69b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 20:23:23.857188 25769 raft_consensus.cc:482] T 1e69a3414cee4903a74610495a89ce1f P 70e535e06c4b430c9ec90f98dbc4f69b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 20:23:23.857311 25758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:23.847093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.53.66:37875 (local address 127.17.53.67:38515)
0929 20:23:23.847310 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0929 20:23:23.847316 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:23:23.847888 (+ 572us) server_negotiation.cc:407] Connection header received
0929 20:23:23.848232 (+ 344us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:23:23.848237 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:23:23.848306 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:23:23.848402 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:23:23.849087 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:23.850228 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:23.854451 (+ 4223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:23.855946 (+ 1495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:23.856122 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:23.856207 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0929 20:23:23.856759 (+ 552us) server_negotiation.cc:299] Negotiation successful
0929 20:23:23.856909 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260929 20:23:23.857311 25769 raft_consensus.cc:3037] T 1e69a3414cee4903a74610495a89ce1f P 70e535e06c4b430c9ec90f98dbc4f69b [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:23:23.857369 25806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:23.846651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.67:38515 (local address 127.17.53.66:37875)
0929 20:23:23.847750 (+ 1099us) negotiation.cc:107] Waiting for socket to connect
0929 20:23:23.847778 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0929 20:23:23.848011 (+ 233us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:23:23.848420 (+ 409us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:23:23.848424 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:23:23.848451 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:23:23.848880 (+ 429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:23:23.848898 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:23:23.850351 (+ 1453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:23:23.850357 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:23:23.852662 (+ 2305us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:23:23.852675 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:23:23.856355 (+ 3680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:23:23.856364 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:23:23.856556 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:23.856595 (+ 39us) client_negotiation.cc:769] Sending connection context
0929 20:23:23.856699 (+ 104us) client_negotiation.cc:240] Negotiation successful
0929 20:23:23.856845 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":937,"mutex_wait_us":89,"thread_start_us":1087,"threads_started":1}
I20260929 20:23:23.858024 25686 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e69a3414cee4903a74610495a89ce1f" candidate_uuid: "70e535e06c4b430c9ec90f98dbc4f69b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f42427cf15e945249ac2061305a6f745" is_pre_election: true
I20260929 20:23:23.858289 25686 raft_consensus.cc:2454] T 1e69a3414cee4903a74610495a89ce1f P f42427cf15e945249ac2061305a6f745 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 70e535e06c4b430c9ec90f98dbc4f69b in term 0.
I20260929 20:23:23.859890 25769 raft_consensus.cc:504] T 1e69a3414cee4903a74610495a89ce1f P 70e535e06c4b430c9ec90f98dbc4f69b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3630496a5db48fda15d65cfcfcf175f" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 41053 } } peers { permanent_uuid: "70e535e06c4b430c9ec90f98dbc4f69b" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 40107 } } peers { permanent_uuid: "f42427cf15e945249ac2061305a6f745" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 38515 } }
I20260929 20:23:23.860536 25769 leader_election.cc:288] T 1e69a3414cee4903a74610495a89ce1f P 70e535e06c4b430c9ec90f98dbc4f69b [CANDIDATE]: Term 1 election: Requested vote from peers f3630496a5db48fda15d65cfcfcf175f (127.17.53.65:41053), f42427cf15e945249ac2061305a6f745 (127.17.53.67:38515)
I20260929 20:23:23.861044 25407 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e69a3414cee4903a74610495a89ce1f" candidate_uuid: "70e535e06c4b430c9ec90f98dbc4f69b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3630496a5db48fda15d65cfcfcf175f"
I20260929 20:23:23.861289 25407 raft_consensus.cc:3037] T 1e69a3414cee4903a74610495a89ce1f P f3630496a5db48fda15d65cfcfcf175f [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:23:23.861500 25686 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1e69a3414cee4903a74610495a89ce1f" candidate_uuid: "70e535e06c4b430c9ec90f98dbc4f69b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f42427cf15e945249ac2061305a6f745"
I20260929 20:23:23.861742 25686 raft_consensus.cc:3037] T 1e69a3414cee4903a74610495a89ce1f P f42427cf15e945249ac2061305a6f745 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:23:23.864830 25407 raft_consensus.cc:2454] T 1e69a3414cee4903a74610495a89ce1f P f3630496a5db48fda15d65cfcfcf175f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 70e535e06c4b430c9ec90f98dbc4f69b in term 1.
I20260929 20:23:23.864857 25686 raft_consensus.cc:2454] T 1e69a3414cee4903a74610495a89ce1f P f42427cf15e945249ac2061305a6f745 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 70e535e06c4b430c9ec90f98dbc4f69b in term 1.
I20260929 20:23:23.865290 25478 leader_election.cc:302] T 1e69a3414cee4903a74610495a89ce1f P 70e535e06c4b430c9ec90f98dbc4f69b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 70e535e06c4b430c9ec90f98dbc4f69b, f3630496a5db48fda15d65cfcfcf175f; no voters:
I20260929 20:23:23.865607 25769 raft_consensus.cc:2781] T 1e69a3414cee4903a74610495a89ce1f P 70e535e06c4b430c9ec90f98dbc4f69b [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:23:23.867353 25769 raft_consensus.cc:686] T 1e69a3414cee4903a74610495a89ce1f P 70e535e06c4b430c9ec90f98dbc4f69b [term 1 LEADER]: Becoming Leader. State: Replica: 70e535e06c4b430c9ec90f98dbc4f69b, State: Running, Role: LEADER
I20260929 20:23:23.867847 25769 consensus_queue.cc:237] T 1e69a3414cee4903a74610495a89ce1f P 70e535e06c4b430c9ec90f98dbc4f69b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3630496a5db48fda15d65cfcfcf175f" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 41053 } } peers { permanent_uuid: "70e535e06c4b430c9ec90f98dbc4f69b" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 40107 } } peers { permanent_uuid: "f42427cf15e945249ac2061305a6f745" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 38515 } }
I20260929 20:23:23.870752 25807 tablet_replica.cc:439] T 1e69a3414cee4903a74610495a89ce1f P 70e535e06c4b430c9ec90f98dbc4f69b: TxnStatusTablet state changed. Reason: New leader 70e535e06c4b430c9ec90f98dbc4f69b. Latest consensus state: current_term: 1 leader_uuid: "70e535e06c4b430c9ec90f98dbc4f69b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3630496a5db48fda15d65cfcfcf175f" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 41053 } } peers { permanent_uuid: "70e535e06c4b430c9ec90f98dbc4f69b" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 40107 } } peers { permanent_uuid: "f42427cf15e945249ac2061305a6f745" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 38515 } } }
I20260929 20:23:23.871135 25807 tablet_replica.cc:442] T 1e69a3414cee4903a74610495a89ce1f P 70e535e06c4b430c9ec90f98dbc4f69b: This TxnStatusTablet replica's current role is: LEADER
I20260929 20:23:23.872408 25249 catalog_manager.cc:5495] T 1e69a3414cee4903a74610495a89ce1f P 70e535e06c4b430c9ec90f98dbc4f69b reported cstate change: term changed from 0 to 1, leader changed from <none> to 70e535e06c4b430c9ec90f98dbc4f69b (127.17.53.66). New cstate: current_term: 1 leader_uuid: "70e535e06c4b430c9ec90f98dbc4f69b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3630496a5db48fda15d65cfcfcf175f" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 41053 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "70e535e06c4b430c9ec90f98dbc4f69b" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 40107 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f42427cf15e945249ac2061305a6f745" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 38515 } health_report { overall_health: UNKNOWN } } }
I20260929 20:23:23.874459 25809 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260929 20:23:23.874693 25809 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260929 20:23:23.922510 25800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:23.914578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.66:40107 (local address 127.0.0.1:45782)
0929 20:23:23.915011 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0929 20:23:23.915034 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0929 20:23:23.915188 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:23:23.915740 (+ 552us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:23:23.915747 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:23:23.915763 (+ 16us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 20:23:23.915958 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:23:23.915972 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:23:23.917444 (+ 1472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:23:23.917450 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:23:23.919162 (+ 1712us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:23:23.919178 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:23:23.920357 (+ 1179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:23:23.920364 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:23:23.920577 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:23.920654 (+ 77us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 20:23:23.921885 (+ 1231us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 20:23:23.921913 (+ 28us) client_negotiation.cc:769] Sending connection context
0929 20:23:23.922007 (+ 94us) client_negotiation.cc:240] Negotiation successful
0929 20:23:23.922181 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":218}
I20260929 20:23:23.922529 25756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:23.914582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45782 (local address 127.17.53.66:40107)
0929 20:23:23.914769 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0929 20:23:23.914776 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:23:23.915135 (+ 359us) server_negotiation.cc:407] Connection header received
0929 20:23:23.915409 (+ 274us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:23:23.915417 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:23:23.915532 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:23:23.915700 (+ 168us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 20:23:23.916113 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:23.917276 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:23.919319 (+ 2043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:23.920003 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:23.920133 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:23.920865 (+ 732us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 20:23:23.921680 (+ 815us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 20:23:23.921829 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0929 20:23:23.922085 (+ 256us) server_negotiation.cc:299] Negotiation successful
0929 20:23:23.922181 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 20:23:23.927803 25407 raft_consensus.cc:1261] T 1e69a3414cee4903a74610495a89ce1f P f3630496a5db48fda15d65cfcfcf175f [term 1 FOLLOWER]: Refusing update from remote peer 70e535e06c4b430c9ec90f98dbc4f69b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:23:23.928076 25686 raft_consensus.cc:1261] T 1e69a3414cee4903a74610495a89ce1f P f42427cf15e945249ac2061305a6f745 [term 1 FOLLOWER]: Refusing update from remote peer 70e535e06c4b430c9ec90f98dbc4f69b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:23:23.929020 25807 consensus_queue.cc:1035] T 1e69a3414cee4903a74610495a89ce1f P 70e535e06c4b430c9ec90f98dbc4f69b [LEADER]: Connected to new peer: Peer: permanent_uuid: "f42427cf15e945249ac2061305a6f745" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 38515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:23:23.929369 25769 consensus_queue.cc:1035] T 1e69a3414cee4903a74610495a89ce1f P 70e535e06c4b430c9ec90f98dbc4f69b [LEADER]: Connected to new peer: Peer: permanent_uuid: "f3630496a5db48fda15d65cfcfcf175f" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 41053 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:23:23.937358 25772 tablet_replica.cc:439] T 1e69a3414cee4903a74610495a89ce1f P f42427cf15e945249ac2061305a6f745: TxnStatusTablet state changed. Reason: New leader 70e535e06c4b430c9ec90f98dbc4f69b. Latest consensus state: current_term: 1 leader_uuid: "70e535e06c4b430c9ec90f98dbc4f69b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3630496a5db48fda15d65cfcfcf175f" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 41053 } } peers { permanent_uuid: "70e535e06c4b430c9ec90f98dbc4f69b" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 40107 } } peers { permanent_uuid: "f42427cf15e945249ac2061305a6f745" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 38515 } } }
I20260929 20:23:23.937701 25772 tablet_replica.cc:442] T 1e69a3414cee4903a74610495a89ce1f P f42427cf15e945249ac2061305a6f745: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 20:23:23.946702 25768 tablet_replica.cc:439] T 1e69a3414cee4903a74610495a89ce1f P f3630496a5db48fda15d65cfcfcf175f: TxnStatusTablet state changed. Reason: New leader 70e535e06c4b430c9ec90f98dbc4f69b. Latest consensus state: current_term: 1 leader_uuid: "70e535e06c4b430c9ec90f98dbc4f69b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3630496a5db48fda15d65cfcfcf175f" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 41053 } } peers { permanent_uuid: "70e535e06c4b430c9ec90f98dbc4f69b" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 40107 } } peers { permanent_uuid: "f42427cf15e945249ac2061305a6f745" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 38515 } } }
I20260929 20:23:23.947050 25768 tablet_replica.cc:442] T 1e69a3414cee4903a74610495a89ce1f P f3630496a5db48fda15d65cfcfcf175f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 20:23:23.948619 25769 tablet_replica.cc:439] T 1e69a3414cee4903a74610495a89ce1f P 70e535e06c4b430c9ec90f98dbc4f69b: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "70e535e06c4b430c9ec90f98dbc4f69b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3630496a5db48fda15d65cfcfcf175f" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 41053 } } peers { permanent_uuid: "70e535e06c4b430c9ec90f98dbc4f69b" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 40107 } } peers { permanent_uuid: "f42427cf15e945249ac2061305a6f745" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 38515 } } }
I20260929 20:23:23.948894 25769 tablet_replica.cc:442] T 1e69a3414cee4903a74610495a89ce1f P 70e535e06c4b430c9ec90f98dbc4f69b: This TxnStatusTablet replica's current role is: LEADER
I20260929 20:23:23.949314 25777 mvcc.cc:205] Tried to move back new op lower bound from 7334762102479757312 to 7334762102246645760. Current Snapshot: MvccSnapshot[applied={T|T < 7334762102479757312 or (T in {7334762102479757312})}]
I20260929 20:23:23.951637 25772 tablet_replica.cc:439] T 1e69a3414cee4903a74610495a89ce1f P f42427cf15e945249ac2061305a6f745: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "70e535e06c4b430c9ec90f98dbc4f69b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3630496a5db48fda15d65cfcfcf175f" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 41053 } } peers { permanent_uuid: "70e535e06c4b430c9ec90f98dbc4f69b" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 40107 } } peers { permanent_uuid: "f42427cf15e945249ac2061305a6f745" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 38515 } } }
I20260929 20:23:23.951906 25772 tablet_replica.cc:442] T 1e69a3414cee4903a74610495a89ce1f P f42427cf15e945249ac2061305a6f745: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 20:23:23.954453 25807 tablet_replica.cc:439] T 1e69a3414cee4903a74610495a89ce1f P 70e535e06c4b430c9ec90f98dbc4f69b: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "70e535e06c4b430c9ec90f98dbc4f69b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3630496a5db48fda15d65cfcfcf175f" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 41053 } } peers { permanent_uuid: "70e535e06c4b430c9ec90f98dbc4f69b" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 40107 } } peers { permanent_uuid: "f42427cf15e945249ac2061305a6f745" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 38515 } } }
I20260929 20:23:23.954717 25807 tablet_replica.cc:442] T 1e69a3414cee4903a74610495a89ce1f P 70e535e06c4b430c9ec90f98dbc4f69b: This TxnStatusTablet replica's current role is: LEADER
I20260929 20:23:23.956182 25768 tablet_replica.cc:439] T 1e69a3414cee4903a74610495a89ce1f P f3630496a5db48fda15d65cfcfcf175f: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "70e535e06c4b430c9ec90f98dbc4f69b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3630496a5db48fda15d65cfcfcf175f" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 41053 } } peers { permanent_uuid: "70e535e06c4b430c9ec90f98dbc4f69b" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 40107 } } peers { permanent_uuid: "f42427cf15e945249ac2061305a6f745" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 38515 } } }
I20260929 20:23:23.956476 25768 tablet_replica.cc:442] T 1e69a3414cee4903a74610495a89ce1f P f3630496a5db48fda15d65cfcfcf175f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260929 20:23:23.973683 25760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:23.965191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46668 (local address 127.17.53.65:41053)
0929 20:23:23.965360 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0929 20:23:23.965366 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:23:23.965386 (+ 20us) server_negotiation.cc:407] Connection header received
0929 20:23:23.965489 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:23:23.965495 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:23:23.965642 (+ 147us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:23:23.965801 (+ 159us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 20:23:23.966403 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:23.967576 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:23.970109 (+ 2533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:23.970783 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:23.970986 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:23.971631 (+ 645us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 20:23:23.972431 (+ 800us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 20:23:23.972584 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0929 20:23:23.973280 (+ 696us) server_negotiation.cc:299] Negotiation successful
0929 20:23:23.973366 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260929 20:23:24.002815 25756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:23.994706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.53.65:35259 (local address 127.17.53.66:40107)
0929 20:23:23.994928 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0929 20:23:23.994935 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:23:23.995618 (+ 683us) server_negotiation.cc:407] Connection header received
0929 20:23:23.995835 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:23:23.995841 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:23:23.995930 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:23:23.996043 (+ 113us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 20:23:23.996645 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:23.997851 (+ 1206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:24.000258 (+ 2407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:24.001122 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:24.001265 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:24.001906 (+ 641us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 20:23:24.002074 (+ 168us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 20:23:24.002174 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0929 20:23:24.002342 (+ 168us) server_negotiation.cc:299] Negotiation successful
0929 20:23:24.002435 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260929 20:23:24.002955 25816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:23.994700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.66:40107 (local address 127.17.53.65:35259)
0929 20:23:23.995478 (+ 778us) negotiation.cc:107] Waiting for socket to connect
0929 20:23:23.995499 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 20:23:23.995678 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:23:23.996081 (+ 403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:23:23.996087 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:23:23.996102 (+ 15us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 20:23:23.996457 (+ 355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:23:23.996489 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:23:23.998019 (+ 1530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:23:23.998026 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:23:24.000078 (+ 2052us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:23:24.000100 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:23:24.001332 (+ 1232us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:23:24.001340 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:23:24.001554 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:24.001648 (+ 94us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 20:23:24.002194 (+ 546us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 20:23:24.002207 (+ 13us) client_negotiation.cc:769] Sending connection context
0929 20:23:24.002306 (+ 99us) client_negotiation.cc:240] Negotiation successful
0929 20:23:24.002494 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":569,"thread_start_us":400,"threads_started":1}
I20260929 20:23:24.015202 25686 raft_consensus.cc:1261] T 6588bacc31ea4864b73f3f626967a1b0 P f42427cf15e945249ac2061305a6f745 [term 1 FOLLOWER]: Refusing update from remote peer f3630496a5db48fda15d65cfcfcf175f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:23:24.015647 25551 raft_consensus.cc:1261] T 6588bacc31ea4864b73f3f626967a1b0 P 70e535e06c4b430c9ec90f98dbc4f69b [term 1 FOLLOWER]: Refusing update from remote peer f3630496a5db48fda15d65cfcfcf175f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:23:24.015893 25793 consensus_queue.cc:1035] T 6588bacc31ea4864b73f3f626967a1b0 P f3630496a5db48fda15d65cfcfcf175f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f42427cf15e945249ac2061305a6f745" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 38515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:23:24.016294 25768 consensus_queue.cc:1035] T 6588bacc31ea4864b73f3f626967a1b0 P f3630496a5db48fda15d65cfcfcf175f [LEADER]: Connected to new peer: Peer: permanent_uuid: "70e535e06c4b430c9ec90f98dbc4f69b" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 40107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:23:24.080930 25760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:24.073860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.53.66:40415 (local address 127.17.53.65:41053)
0929 20:23:24.074085 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0929 20:23:24.074092 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:23:24.074692 (+ 600us) server_negotiation.cc:407] Connection header received
0929 20:23:24.074869 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:23:24.074876 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:23:24.074964 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:23:24.075065 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 20:23:24.075588 (+ 523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:24.076715 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:24.079044 (+ 2329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:24.079501 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:24.079631 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:24.080169 (+ 538us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 20:23:24.080301 (+ 132us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 20:23:24.080409 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0929 20:23:24.080520 (+ 111us) server_negotiation.cc:299] Negotiation successful
0929 20:23:24.080598 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 20:23:24.081051 25822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:24.073878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.65:41053 (local address 127.17.53.66:40415)
0929 20:23:24.074598 (+ 720us) negotiation.cc:107] Waiting for socket to connect
0929 20:23:24.074615 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 20:23:24.074737 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:23:24.075117 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:23:24.075120 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:23:24.075134 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0929 20:23:24.075433 (+ 299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:23:24.075441 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:23:24.076895 (+ 1454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:23:24.076901 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:23:24.078906 (+ 2005us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:23:24.078922 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:23:24.079668 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:23:24.079675 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:23:24.079900 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:24.079976 (+ 76us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0929 20:23:24.080409 (+ 433us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0929 20:23:24.080423 (+ 14us) client_negotiation.cc:769] Sending connection context
0929 20:23:24.080501 (+ 78us) client_negotiation.cc:240] Negotiation successful
0929 20:23:24.080637 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":528,"thread_start_us":384,"threads_started":1}
I20260929 20:23:24.126560 25250 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57850:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 20:23:24.126873 25250 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57850:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 20:23:24.130569 25250 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d27fa57b2ddf4b438a0d3fd3a5271ac6: Sending DeleteTablet for 3 replicas of tablet 6588bacc31ea4864b73f3f626967a1b0
I20260929 20:23:24.131705 25666 tablet_service.cc:1514] Processing DeleteTablet for tablet 6588bacc31ea4864b73f3f626967a1b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:23:24 UTC) from {username='oryx', principal='oryx/127.17.53.126@KRBTEST.COM'} at 127.0.0.1:47892
I20260929 20:23:24.131784 25387 tablet_service.cc:1514] Processing DeleteTablet for tablet 6588bacc31ea4864b73f3f626967a1b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:23:24 UTC) from {username='oryx', principal='oryx/127.17.53.126@KRBTEST.COM'} at 127.0.0.1:46666
I20260929 20:23:24.132162 25527 tablet_service.cc:1514] Processing DeleteTablet for tablet 6588bacc31ea4864b73f3f626967a1b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:23:24 UTC) from {username='oryx', principal='oryx/127.17.53.126@KRBTEST.COM'} at 127.0.0.1:45766
I20260929 20:23:24.132387 25826 tablet_replica.cc:330] T 6588bacc31ea4864b73f3f626967a1b0 P f42427cf15e945249ac2061305a6f745: stopping tablet replica
I20260929 20:23:24.132627 25826 raft_consensus.cc:2229] T 6588bacc31ea4864b73f3f626967a1b0 P f42427cf15e945249ac2061305a6f745 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 20:23:24.132912 25826 raft_consensus.cc:2258] T 6588bacc31ea4864b73f3f626967a1b0 P f42427cf15e945249ac2061305a6f745 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:23:24.134167 25826 ts_tablet_manager.cc:1905] T 6588bacc31ea4864b73f3f626967a1b0 P f42427cf15e945249ac2061305a6f745: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:23:24.142508 25826 ts_tablet_manager.cc:1918] T 6588bacc31ea4864b73f3f626967a1b0 P f42427cf15e945249ac2061305a6f745: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 20:23:24.142714 25826 log.cc:1196] T 6588bacc31ea4864b73f3f626967a1b0 P f42427cf15e945249ac2061305a6f745: Deleting WAL directory at /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790713339161492-17621-0/minicluster-data/ts-2/wal/wals/6588bacc31ea4864b73f3f626967a1b0
I20260929 20:23:24.143321 25826 ts_tablet_manager.cc:1939] T 6588bacc31ea4864b73f3f626967a1b0 P f42427cf15e945249ac2061305a6f745: Deleting consensus metadata
I20260929 20:23:24.144572 25234 catalog_manager.cc:4841] TS f42427cf15e945249ac2061305a6f745 (127.17.53.67:38515): tablet 6588bacc31ea4864b73f3f626967a1b0 (table test-table [id=8bd8450d209d483a8b9121b70b9b7809]) successfully deleted
I20260929 20:23:24.146090 25828 tablet_replica.cc:330] T 6588bacc31ea4864b73f3f626967a1b0 P 70e535e06c4b430c9ec90f98dbc4f69b: stopping tablet replica
I20260929 20:23:24.146332 25828 raft_consensus.cc:2229] T 6588bacc31ea4864b73f3f626967a1b0 P 70e535e06c4b430c9ec90f98dbc4f69b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 20:23:24.146672 25828 raft_consensus.cc:2258] T 6588bacc31ea4864b73f3f626967a1b0 P 70e535e06c4b430c9ec90f98dbc4f69b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:23:24.147580 25827 tablet_replica.cc:330] T 6588bacc31ea4864b73f3f626967a1b0 P f3630496a5db48fda15d65cfcfcf175f: stopping tablet replica
I20260929 20:23:24.147814 25827 raft_consensus.cc:2229] T 6588bacc31ea4864b73f3f626967a1b0 P f3630496a5db48fda15d65cfcfcf175f [term 1 LEADER]: Raft consensus shutting down.
I20260929 20:23:24.148362 25827 raft_consensus.cc:2258] T 6588bacc31ea4864b73f3f626967a1b0 P f3630496a5db48fda15d65cfcfcf175f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:23:24.149178 25828 ts_tablet_manager.cc:1905] T 6588bacc31ea4864b73f3f626967a1b0 P 70e535e06c4b430c9ec90f98dbc4f69b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:23:24.149459 25827 ts_tablet_manager.cc:1905] T 6588bacc31ea4864b73f3f626967a1b0 P f3630496a5db48fda15d65cfcfcf175f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:23:24.157826 25827 ts_tablet_manager.cc:1918] T 6588bacc31ea4864b73f3f626967a1b0 P f3630496a5db48fda15d65cfcfcf175f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 20:23:24.158109 25827 log.cc:1196] T 6588bacc31ea4864b73f3f626967a1b0 P f3630496a5db48fda15d65cfcfcf175f: Deleting WAL directory at /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790713339161492-17621-0/minicluster-data/ts-0/wal/wals/6588bacc31ea4864b73f3f626967a1b0
I20260929 20:23:24.158713 25827 ts_tablet_manager.cc:1939] T 6588bacc31ea4864b73f3f626967a1b0 P f3630496a5db48fda15d65cfcfcf175f: Deleting consensus metadata
I20260929 20:23:24.160070 25235 catalog_manager.cc:4841] TS f3630496a5db48fda15d65cfcfcf175f (127.17.53.65:41053): tablet 6588bacc31ea4864b73f3f626967a1b0 (table test-table [id=8bd8450d209d483a8b9121b70b9b7809]) successfully deleted
I20260929 20:23:24.168870 25828 ts_tablet_manager.cc:1918] T 6588bacc31ea4864b73f3f626967a1b0 P 70e535e06c4b430c9ec90f98dbc4f69b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 20:23:24.169190 25828 log.cc:1196] T 6588bacc31ea4864b73f3f626967a1b0 P 70e535e06c4b430c9ec90f98dbc4f69b: Deleting WAL directory at /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790713339161492-17621-0/minicluster-data/ts-1/wal/wals/6588bacc31ea4864b73f3f626967a1b0
I20260929 20:23:24.169948 25828 ts_tablet_manager.cc:1939] T 6588bacc31ea4864b73f3f626967a1b0 P 70e535e06c4b430c9ec90f98dbc4f69b: Deleting consensus metadata
I20260929 20:23:24.171888 25235 catalog_manager.cc:4841] TS 70e535e06c4b430c9ec90f98dbc4f69b (127.17.53.66:40107): tablet 6588bacc31ea4864b73f3f626967a1b0 (table test-table [id=8bd8450d209d483a8b9121b70b9b7809]) successfully deleted
Sep 29 20:23:24 dist-test-slave-zbnc krb5kdc[25205](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713404, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 29 20:23:24 dist-test-slave-zbnc krb5kdc[25205](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.17.53.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 20:23:24 dist-test-slave-zbnc krb5kdc[25205](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.17.53.126@KRBTEST.COM, Server not found in Kerberos database
W20260929 20:23:24.212277 25801 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:23:24.196903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57870 (local address 127.17.53.126:36941)
0929 20:23:24.197155 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0929 20:23:24.197165 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 20:23:24.197770 (+ 605us) server_negotiation.cc:407] Connection header received
0929 20:23:24.198039 (+ 269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:23:24.198045 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:23:24.198155 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:23:24.198261 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:23:24.200331 (+ 2070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:24.201591 (+ 1260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:24.203719 (+ 2128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:24.204491 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:24.204591 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:24.211976 (+ 7385us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57870: BlockingRecv error: recv got EOF from 127.0.0.1:57870 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
Sep 29 20:23:24 dist-test-slave-zbnc krb5kdc[25205](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713404, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.17.53.126@KRBTEST.COM
I20260929 20:23:24.266840 25801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:24.248880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57874 (local address 127.17.53.126:36941)
0929 20:23:24.249097 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0929 20:23:24.249108 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 20:23:24.249667 (+ 559us) server_negotiation.cc:407] Connection header received
0929 20:23:24.249927 (+ 260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:23:24.249935 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:23:24.250030 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:23:24.250139 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:23:24.251950 (+ 1811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:24.252760 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:24.254138 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:24.254621 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:24.254771 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:24.259188 (+ 4417us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:23:24.259194 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:23:24.259197 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:23:24.259241 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:23:24.263245 (+ 4004us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:23:24.264307 (+ 1062us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:23:24.264314 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:23:24.264318 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:23:24.264486 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:23:24.264982 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:23:24.264988 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:23:24.264992 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:23:24.265384 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:23:24.265688 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0929 20:23:24.266084 (+ 396us) server_negotiation.cc:299] Negotiation successful
0929 20:23:24.266352 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260929 20:23:24.271118 25250 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57874:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 20:23:24.271764 25250 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 20:23:24.278733 25666 tablet_service.cc:1467] Processing CreateTablet for tablet 8e1b1fc42fc94152bd1d6b4cf3c513b8 (DEFAULT_TABLE table=test-table [id=2c5fd3a3d24a4b2d96299f333ab04d4d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:23:24.278765 25387 tablet_service.cc:1467] Processing CreateTablet for tablet 8e1b1fc42fc94152bd1d6b4cf3c513b8 (DEFAULT_TABLE table=test-table [id=2c5fd3a3d24a4b2d96299f333ab04d4d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:23:24.278995 25527 tablet_service.cc:1467] Processing CreateTablet for tablet 8e1b1fc42fc94152bd1d6b4cf3c513b8 (DEFAULT_TABLE table=test-table [id=2c5fd3a3d24a4b2d96299f333ab04d4d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:23:24.279248 25666 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e1b1fc42fc94152bd1d6b4cf3c513b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:23:24.279251 25387 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e1b1fc42fc94152bd1d6b4cf3c513b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:23:24.279382 25527 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8e1b1fc42fc94152bd1d6b4cf3c513b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:23:24.285066 25841 tablet_bootstrap.cc:492] T 8e1b1fc42fc94152bd1d6b4cf3c513b8 P f42427cf15e945249ac2061305a6f745: Bootstrap starting.
I20260929 20:23:24.286110 25842 tablet_bootstrap.cc:492] T 8e1b1fc42fc94152bd1d6b4cf3c513b8 P 70e535e06c4b430c9ec90f98dbc4f69b: Bootstrap starting.
I20260929 20:23:24.286708 25843 tablet_bootstrap.cc:492] T 8e1b1fc42fc94152bd1d6b4cf3c513b8 P f3630496a5db48fda15d65cfcfcf175f: Bootstrap starting.
I20260929 20:23:24.286932 25841 tablet_bootstrap.cc:654] T 8e1b1fc42fc94152bd1d6b4cf3c513b8 P f42427cf15e945249ac2061305a6f745: Neither blocks nor log segments found. Creating new log.
I20260929 20:23:24.293951 25843 tablet_bootstrap.cc:654] T 8e1b1fc42fc94152bd1d6b4cf3c513b8 P f3630496a5db48fda15d65cfcfcf175f: Neither blocks nor log segments found. Creating new log.
I20260929 20:23:24.295137 25842 tablet_bootstrap.cc:654] T 8e1b1fc42fc94152bd1d6b4cf3c513b8 P 70e535e06c4b430c9ec90f98dbc4f69b: Neither blocks nor log segments found. Creating new log.
I20260929 20:23:24.300199 25843 tablet_bootstrap.cc:492] T 8e1b1fc42fc94152bd1d6b4cf3c513b8 P f3630496a5db48fda15d65cfcfcf175f: No bootstrap required, opened a new log
I20260929 20:23:24.300423 25843 ts_tablet_manager.cc:1397] T 8e1b1fc42fc94152bd1d6b4cf3c513b8 P f3630496a5db48fda15d65cfcfcf175f: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.000s
I20260929 20:23:24.301417 25841 tablet_bootstrap.cc:492] T 8e1b1fc42fc94152bd1d6b4cf3c513b8 P f42427cf15e945249ac2061305a6f745: No bootstrap required, opened a new log
I20260929 20:23:24.301367 25843 raft_consensus.cc:348] T 8e1b1fc42fc94152bd1d6b4cf3c513b8 P f3630496a5db48fda15d65cfcfcf175f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3630496a5db48fda15d65cfcfcf175f" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 41053 } } peers { permanent_uuid: "70e535e06c4b430c9ec90f98dbc4f69b" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 40107 } } peers { permanent_uuid: "f42427cf15e945249ac2061305a6f745" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 38515 } }
I20260929 20:23:24.301681 25843 raft_consensus.cc:374] T 8e1b1fc42fc94152bd1d6b4cf3c513b8 P f3630496a5db48fda15d65cfcfcf175f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:23:24.301779 25841 ts_tablet_manager.cc:1397] T 8e1b1fc42fc94152bd1d6b4cf3c513b8 P f42427cf15e945249ac2061305a6f745: Time spent bootstrapping tablet: real 0.017s user 0.004s sys 0.009s
I20260929 20:23:24.301827 25843 raft_consensus.cc:729] T 8e1b1fc42fc94152bd1d6b4cf3c513b8 P f3630496a5db48fda15d65cfcfcf175f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3630496a5db48fda15d65cfcfcf175f, State: Initialized, Role: FOLLOWER
I20260929 20:23:24.302143 25843 consensus_queue.cc:260] T 8e1b1fc42fc94152bd1d6b4cf3c513b8 P f3630496a5db48fda15d65cfcfcf175f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3630496a5db48fda15d65cfcfcf175f" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 41053 } } peers { permanent_uuid: "70e535e06c4b430c9ec90f98dbc4f69b" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 40107 } } peers { permanent_uuid: "f42427cf15e945249ac2061305a6f745" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 38515 } }
I20260929 20:23:24.302755 25843 ts_tablet_manager.cc:1428] T 8e1b1fc42fc94152bd1d6b4cf3c513b8 P f3630496a5db48fda15d65cfcfcf175f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260929 20:23:24.303606 25841 raft_consensus.cc:348] T 8e1b1fc42fc94152bd1d6b4cf3c513b8 P f42427cf15e945249ac2061305a6f745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3630496a5db48fda15d65cfcfcf175f" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 41053 } } peers { permanent_uuid: "70e535e06c4b430c9ec90f98dbc4f69b" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 40107 } } peers { permanent_uuid: "f42427cf15e945249ac2061305a6f745" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 38515 } }
I20260929 20:23:24.303927 25841 raft_consensus.cc:374] T 8e1b1fc42fc94152bd1d6b4cf3c513b8 P f42427cf15e945249ac2061305a6f745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:23:24.304068 25841 raft_consensus.cc:729] T 8e1b1fc42fc94152bd1d6b4cf3c513b8 P f42427cf15e945249ac2061305a6f745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f42427cf15e945249ac2061305a6f745, State: Initialized, Role: FOLLOWER
I20260929 20:23:24.304401 25841 consensus_queue.cc:260] T 8e1b1fc42fc94152bd1d6b4cf3c513b8 P f42427cf15e945249ac2061305a6f745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3630496a5db48fda15d65cfcfcf175f" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 41053 } } peers { permanent_uuid: "70e535e06c4b430c9ec90f98dbc4f69b" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 40107 } } peers { permanent_uuid: "f42427cf15e945249ac2061305a6f745" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 38515 } }
I20260929 20:23:24.308243 25841 ts_tablet_manager.cc:1428] T 8e1b1fc42fc94152bd1d6b4cf3c513b8 P f42427cf15e945249ac2061305a6f745: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20260929 20:23:24.309509 25842 tablet_bootstrap.cc:492] T 8e1b1fc42fc94152bd1d6b4cf3c513b8 P 70e535e06c4b430c9ec90f98dbc4f69b: No bootstrap required, opened a new log
I20260929 20:23:24.310415 25842 ts_tablet_manager.cc:1397] T 8e1b1fc42fc94152bd1d6b4cf3c513b8 P 70e535e06c4b430c9ec90f98dbc4f69b: Time spent bootstrapping tablet: real 0.024s user 0.005s sys 0.000s
I20260929 20:23:24.311254 25842 raft_consensus.cc:348] T 8e1b1fc42fc94152bd1d6b4cf3c513b8 P 70e535e06c4b430c9ec90f98dbc4f69b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3630496a5db48fda15d65cfcfcf175f" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 41053 } } peers { permanent_uuid: "70e535e06c4b430c9ec90f98dbc4f69b" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 40107 } } peers { permanent_uuid: "f42427cf15e945249ac2061305a6f745" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 38515 } }
I20260929 20:23:24.311491 25842 raft_consensus.cc:374] T 8e1b1fc42fc94152bd1d6b4cf3c513b8 P 70e535e06c4b430c9ec90f98dbc4f69b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:23:24.311611 25842 raft_consensus.cc:729] T 8e1b1fc42fc94152bd1d6b4cf3c513b8 P 70e535e06c4b430c9ec90f98dbc4f69b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 70e535e06c4b430c9ec90f98dbc4f69b, State: Initialized, Role: FOLLOWER
I20260929 20:23:24.311858 25842 consensus_queue.cc:260] T 8e1b1fc42fc94152bd1d6b4cf3c513b8 P 70e535e06c4b430c9ec90f98dbc4f69b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3630496a5db48fda15d65cfcfcf175f" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 41053 } } peers { permanent_uuid: "70e535e06c4b430c9ec90f98dbc4f69b" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 40107 } } peers { permanent_uuid: "f42427cf15e945249ac2061305a6f745" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 38515 } }
I20260929 20:23:24.313817 25842 ts_tablet_manager.cc:1428] T 8e1b1fc42fc94152bd1d6b4cf3c513b8 P 70e535e06c4b430c9ec90f98dbc4f69b: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260929 20:23:24.651268 25817 raft_consensus.cc:482] T 8e1b1fc42fc94152bd1d6b4cf3c513b8 P 70e535e06c4b430c9ec90f98dbc4f69b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 20:23:24.651456 25817 raft_consensus.cc:504] T 8e1b1fc42fc94152bd1d6b4cf3c513b8 P 70e535e06c4b430c9ec90f98dbc4f69b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3630496a5db48fda15d65cfcfcf175f" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 41053 } } peers { permanent_uuid: "70e535e06c4b430c9ec90f98dbc4f69b" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 40107 } } peers { permanent_uuid: "f42427cf15e945249ac2061305a6f745" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 38515 } }
I20260929 20:23:24.652118 25817 leader_election.cc:288] T 8e1b1fc42fc94152bd1d6b4cf3c513b8 P 70e535e06c4b430c9ec90f98dbc4f69b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f3630496a5db48fda15d65cfcfcf175f (127.17.53.65:41053), f42427cf15e945249ac2061305a6f745 (127.17.53.67:38515)
I20260929 20:23:24.652729 25685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e1b1fc42fc94152bd1d6b4cf3c513b8" candidate_uuid: "70e535e06c4b430c9ec90f98dbc4f69b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f42427cf15e945249ac2061305a6f745" is_pre_election: true
I20260929 20:23:24.652766 25406 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e1b1fc42fc94152bd1d6b4cf3c513b8" candidate_uuid: "70e535e06c4b430c9ec90f98dbc4f69b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3630496a5db48fda15d65cfcfcf175f" is_pre_election: true
I20260929 20:23:24.653012 25406 raft_consensus.cc:2454] T 8e1b1fc42fc94152bd1d6b4cf3c513b8 P f3630496a5db48fda15d65cfcfcf175f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 70e535e06c4b430c9ec90f98dbc4f69b in term 0.
I20260929 20:23:24.653020 25685 raft_consensus.cc:2454] T 8e1b1fc42fc94152bd1d6b4cf3c513b8 P f42427cf15e945249ac2061305a6f745 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 70e535e06c4b430c9ec90f98dbc4f69b in term 0.
I20260929 20:23:24.653509 25477 leader_election.cc:302] T 8e1b1fc42fc94152bd1d6b4cf3c513b8 P 70e535e06c4b430c9ec90f98dbc4f69b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 70e535e06c4b430c9ec90f98dbc4f69b, f42427cf15e945249ac2061305a6f745; no voters:
I20260929 20:23:24.653801 25817 raft_consensus.cc:2781] T 8e1b1fc42fc94152bd1d6b4cf3c513b8 P 70e535e06c4b430c9ec90f98dbc4f69b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 20:23:24.653925 25817 raft_consensus.cc:482] T 8e1b1fc42fc94152bd1d6b4cf3c513b8 P 70e535e06c4b430c9ec90f98dbc4f69b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 20:23:24.654034 25817 raft_consensus.cc:3037] T 8e1b1fc42fc94152bd1d6b4cf3c513b8 P 70e535e06c4b430c9ec90f98dbc4f69b [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:23:24.656026 25817 raft_consensus.cc:504] T 8e1b1fc42fc94152bd1d6b4cf3c513b8 P 70e535e06c4b430c9ec90f98dbc4f69b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3630496a5db48fda15d65cfcfcf175f" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 41053 } } peers { permanent_uuid: "70e535e06c4b430c9ec90f98dbc4f69b" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 40107 } } peers { permanent_uuid: "f42427cf15e945249ac2061305a6f745" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 38515 } }
I20260929 20:23:24.656538 25817 leader_election.cc:288] T 8e1b1fc42fc94152bd1d6b4cf3c513b8 P 70e535e06c4b430c9ec90f98dbc4f69b [CANDIDATE]: Term 1 election: Requested vote from peers f3630496a5db48fda15d65cfcfcf175f (127.17.53.65:41053), f42427cf15e945249ac2061305a6f745 (127.17.53.67:38515)
I20260929 20:23:24.657074 25406 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e1b1fc42fc94152bd1d6b4cf3c513b8" candidate_uuid: "70e535e06c4b430c9ec90f98dbc4f69b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3630496a5db48fda15d65cfcfcf175f"
I20260929 20:23:24.657074 25685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8e1b1fc42fc94152bd1d6b4cf3c513b8" candidate_uuid: "70e535e06c4b430c9ec90f98dbc4f69b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f42427cf15e945249ac2061305a6f745"
I20260929 20:23:24.657281 25406 raft_consensus.cc:3037] T 8e1b1fc42fc94152bd1d6b4cf3c513b8 P f3630496a5db48fda15d65cfcfcf175f [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:23:24.657294 25685 raft_consensus.cc:3037] T 8e1b1fc42fc94152bd1d6b4cf3c513b8 P f42427cf15e945249ac2061305a6f745 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:23:24.660151 25406 raft_consensus.cc:2454] T 8e1b1fc42fc94152bd1d6b4cf3c513b8 P f3630496a5db48fda15d65cfcfcf175f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 70e535e06c4b430c9ec90f98dbc4f69b in term 1.
I20260929 20:23:24.660175 25685 raft_consensus.cc:2454] T 8e1b1fc42fc94152bd1d6b4cf3c513b8 P f42427cf15e945249ac2061305a6f745 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 70e535e06c4b430c9ec90f98dbc4f69b in term 1.
I20260929 20:23:24.660570 25477 leader_election.cc:302] T 8e1b1fc42fc94152bd1d6b4cf3c513b8 P 70e535e06c4b430c9ec90f98dbc4f69b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 70e535e06c4b430c9ec90f98dbc4f69b, f3630496a5db48fda15d65cfcfcf175f, f42427cf15e945249ac2061305a6f745; no voters:
I20260929 20:23:24.660882 25817 raft_consensus.cc:2781] T 8e1b1fc42fc94152bd1d6b4cf3c513b8 P 70e535e06c4b430c9ec90f98dbc4f69b [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:23:24.661235 25817 raft_consensus.cc:686] T 8e1b1fc42fc94152bd1d6b4cf3c513b8 P 70e535e06c4b430c9ec90f98dbc4f69b [term 1 LEADER]: Becoming Leader. State: Replica: 70e535e06c4b430c9ec90f98dbc4f69b, State: Running, Role: LEADER
I20260929 20:23:24.661509 25817 consensus_queue.cc:237] T 8e1b1fc42fc94152bd1d6b4cf3c513b8 P 70e535e06c4b430c9ec90f98dbc4f69b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3630496a5db48fda15d65cfcfcf175f" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 41053 } } peers { permanent_uuid: "70e535e06c4b430c9ec90f98dbc4f69b" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 40107 } } peers { permanent_uuid: "f42427cf15e945249ac2061305a6f745" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 38515 } }
I20260929 20:23:24.664179 25249 catalog_manager.cc:5495] T 8e1b1fc42fc94152bd1d6b4cf3c513b8 P 70e535e06c4b430c9ec90f98dbc4f69b reported cstate change: term changed from 0 to 1, leader changed from <none> to 70e535e06c4b430c9ec90f98dbc4f69b (127.17.53.66). New cstate: current_term: 1 leader_uuid: "70e535e06c4b430c9ec90f98dbc4f69b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3630496a5db48fda15d65cfcfcf175f" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 41053 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "70e535e06c4b430c9ec90f98dbc4f69b" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 40107 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f42427cf15e945249ac2061305a6f745" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 38515 } health_report { overall_health: UNKNOWN } } }
I20260929 20:23:24.748436 25850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:24.743633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45788 (local address 127.17.53.66:40107)
0929 20:23:24.744052 (+ 419us) server_negotiation.cc:206] Beginning negotiation
0929 20:23:24.744057 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:23:24.744073 (+ 16us) server_negotiation.cc:407] Connection header received
0929 20:23:24.744131 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:23:24.744135 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:23:24.744205 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:23:24.744284 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 20:23:24.744687 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:24.745371 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:24.746692 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:24.747167 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:24.747259 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:24.747738 (+ 479us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 20:23:24.747835 (+ 97us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 20:23:24.747903 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0929 20:23:24.748015 (+ 112us) server_negotiation.cc:299] Negotiation successful
0929 20:23:24.748075 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":308,"thread_start_us":210,"threads_started":1}
I20260929 20:23:24.751690 25406 raft_consensus.cc:1261] T 8e1b1fc42fc94152bd1d6b4cf3c513b8 P f3630496a5db48fda15d65cfcfcf175f [term 1 FOLLOWER]: Refusing update from remote peer 70e535e06c4b430c9ec90f98dbc4f69b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:23:24.751751 25685 raft_consensus.cc:1261] T 8e1b1fc42fc94152bd1d6b4cf3c513b8 P f42427cf15e945249ac2061305a6f745 [term 1 FOLLOWER]: Refusing update from remote peer 70e535e06c4b430c9ec90f98dbc4f69b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:23:24.752383 25847 consensus_queue.cc:1035] T 8e1b1fc42fc94152bd1d6b4cf3c513b8 P 70e535e06c4b430c9ec90f98dbc4f69b [LEADER]: Connected to new peer: Peer: permanent_uuid: "f3630496a5db48fda15d65cfcfcf175f" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 41053 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:23:24.752723 25817 consensus_queue.cc:1035] T 8e1b1fc42fc94152bd1d6b4cf3c513b8 P 70e535e06c4b430c9ec90f98dbc4f69b [LEADER]: Connected to new peer: Peer: permanent_uuid: "f42427cf15e945249ac2061305a6f745" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 38515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:23:24.776062 25250 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57874:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 20:23:24.776279 25250 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57874:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 20:23:24.779299 25250 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d27fa57b2ddf4b438a0d3fd3a5271ac6: Sending DeleteTablet for 3 replicas of tablet 8e1b1fc42fc94152bd1d6b4cf3c513b8
I20260929 20:23:24.780215 25387 tablet_service.cc:1514] Processing DeleteTablet for tablet 8e1b1fc42fc94152bd1d6b4cf3c513b8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:23:24 UTC) from {username='oryx', principal='oryx/127.17.53.126@KRBTEST.COM'} at 127.0.0.1:46666
I20260929 20:23:24.780212 25666 tablet_service.cc:1514] Processing DeleteTablet for tablet 8e1b1fc42fc94152bd1d6b4cf3c513b8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:23:24 UTC) from {username='oryx', principal='oryx/127.17.53.126@KRBTEST.COM'} at 127.0.0.1:47892
I20260929 20:23:24.780649 25527 tablet_service.cc:1514] Processing DeleteTablet for tablet 8e1b1fc42fc94152bd1d6b4cf3c513b8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:23:24 UTC) from {username='oryx', principal='oryx/127.17.53.126@KRBTEST.COM'} at 127.0.0.1:45766
I20260929 20:23:24.781352 25864 tablet_replica.cc:330] T 8e1b1fc42fc94152bd1d6b4cf3c513b8 P 70e535e06c4b430c9ec90f98dbc4f69b: stopping tablet replica
I20260929 20:23:24.781629 25864 raft_consensus.cc:2229] T 8e1b1fc42fc94152bd1d6b4cf3c513b8 P 70e535e06c4b430c9ec90f98dbc4f69b [term 1 LEADER]: Raft consensus shutting down.
I20260929 20:23:24.781754 25249 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57874:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260929 20:23:24.782298 25864 raft_consensus.cc:2258] T 8e1b1fc42fc94152bd1d6b4cf3c513b8 P 70e535e06c4b430c9ec90f98dbc4f69b [term 1 FOLLOWER]: Raft consensus is shut down!
W20260929 20:23:24.782589 25249 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 20:23:24.783478 25864 ts_tablet_manager.cc:1905] T 8e1b1fc42fc94152bd1d6b4cf3c513b8 P 70e535e06c4b430c9ec90f98dbc4f69b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:23:24.790781 25864 ts_tablet_manager.cc:1918] T 8e1b1fc42fc94152bd1d6b4cf3c513b8 P 70e535e06c4b430c9ec90f98dbc4f69b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 20:23:24.790979 25864 log.cc:1196] T 8e1b1fc42fc94152bd1d6b4cf3c513b8 P 70e535e06c4b430c9ec90f98dbc4f69b: Deleting WAL directory at /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790713339161492-17621-0/minicluster-data/ts-1/wal/wals/8e1b1fc42fc94152bd1d6b4cf3c513b8
I20260929 20:23:24.791495 25862 tablet_replica.cc:330] T 8e1b1fc42fc94152bd1d6b4cf3c513b8 P f3630496a5db48fda15d65cfcfcf175f: stopping tablet replica
I20260929 20:23:24.791549 25864 ts_tablet_manager.cc:1939] T 8e1b1fc42fc94152bd1d6b4cf3c513b8 P 70e535e06c4b430c9ec90f98dbc4f69b: Deleting consensus metadata
I20260929 20:23:24.792235 25862 raft_consensus.cc:2229] T 8e1b1fc42fc94152bd1d6b4cf3c513b8 P f3630496a5db48fda15d65cfcfcf175f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 20:23:24.792712 25235 catalog_manager.cc:4841] TS 70e535e06c4b430c9ec90f98dbc4f69b (127.17.53.66:40107): tablet 8e1b1fc42fc94152bd1d6b4cf3c513b8 (table test-table [id=2c5fd3a3d24a4b2d96299f333ab04d4d]) successfully deleted
I20260929 20:23:24.793045 25862 raft_consensus.cc:2258] T 8e1b1fc42fc94152bd1d6b4cf3c513b8 P f3630496a5db48fda15d65cfcfcf175f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:23:24.794152 25386 tablet_service.cc:1467] Processing CreateTablet for tablet 205f72f6f84945a5b8315b8123fa79e0 (DEFAULT_TABLE table=test-table [id=24feb2db33384ee186c4a7d6dcdccfda]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:23:24.794188 25665 tablet_service.cc:1467] Processing CreateTablet for tablet 205f72f6f84945a5b8315b8123fa79e0 (DEFAULT_TABLE table=test-table [id=24feb2db33384ee186c4a7d6dcdccfda]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:23:24.794433 25862 ts_tablet_manager.cc:1905] T 8e1b1fc42fc94152bd1d6b4cf3c513b8 P f3630496a5db48fda15d65cfcfcf175f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:23:24.794596 25386 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 205f72f6f84945a5b8315b8123fa79e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:23:24.794662 25665 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 205f72f6f84945a5b8315b8123fa79e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:23:24.795310 25863 tablet_replica.cc:330] T 8e1b1fc42fc94152bd1d6b4cf3c513b8 P f42427cf15e945249ac2061305a6f745: stopping tablet replica
I20260929 20:23:24.795490 25863 raft_consensus.cc:2229] T 8e1b1fc42fc94152bd1d6b4cf3c513b8 P f42427cf15e945249ac2061305a6f745 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 20:23:24.795773 25863 raft_consensus.cc:2258] T 8e1b1fc42fc94152bd1d6b4cf3c513b8 P f42427cf15e945249ac2061305a6f745 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:23:24.799223 25527 tablet_service.cc:1467] Processing CreateTablet for tablet 205f72f6f84945a5b8315b8123fa79e0 (DEFAULT_TABLE table=test-table [id=24feb2db33384ee186c4a7d6dcdccfda]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:23:24.799701 25527 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 205f72f6f84945a5b8315b8123fa79e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:23:24.802376 25862 ts_tablet_manager.cc:1918] T 8e1b1fc42fc94152bd1d6b4cf3c513b8 P f3630496a5db48fda15d65cfcfcf175f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 20:23:24.802538 25862 log.cc:1196] T 8e1b1fc42fc94152bd1d6b4cf3c513b8 P f3630496a5db48fda15d65cfcfcf175f: Deleting WAL directory at /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790713339161492-17621-0/minicluster-data/ts-0/wal/wals/8e1b1fc42fc94152bd1d6b4cf3c513b8
I20260929 20:23:24.803824 25863 ts_tablet_manager.cc:1905] T 8e1b1fc42fc94152bd1d6b4cf3c513b8 P f42427cf15e945249ac2061305a6f745: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:23:24.806950 25842 tablet_bootstrap.cc:492] T 205f72f6f84945a5b8315b8123fa79e0 P 70e535e06c4b430c9ec90f98dbc4f69b: Bootstrap starting.
I20260929 20:23:24.808619 25842 tablet_bootstrap.cc:654] T 205f72f6f84945a5b8315b8123fa79e0 P 70e535e06c4b430c9ec90f98dbc4f69b: Neither blocks nor log segments found. Creating new log.
I20260929 20:23:24.809860 25841 tablet_bootstrap.cc:492] T 205f72f6f84945a5b8315b8123fa79e0 P f42427cf15e945249ac2061305a6f745: Bootstrap starting.
I20260929 20:23:24.811796 25863 ts_tablet_manager.cc:1918] T 8e1b1fc42fc94152bd1d6b4cf3c513b8 P f42427cf15e945249ac2061305a6f745: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 20:23:24.812052 25862 ts_tablet_manager.cc:1939] T 8e1b1fc42fc94152bd1d6b4cf3c513b8 P f3630496a5db48fda15d65cfcfcf175f: Deleting consensus metadata
I20260929 20:23:24.812119 25841 tablet_bootstrap.cc:654] T 205f72f6f84945a5b8315b8123fa79e0 P f42427cf15e945249ac2061305a6f745: Neither blocks nor log segments found. Creating new log.
I20260929 20:23:24.813616 25235 catalog_manager.cc:4841] TS f3630496a5db48fda15d65cfcfcf175f (127.17.53.65:41053): tablet 8e1b1fc42fc94152bd1d6b4cf3c513b8 (table test-table [id=2c5fd3a3d24a4b2d96299f333ab04d4d]) successfully deleted
I20260929 20:23:24.814733 25863 log.cc:1196] T 8e1b1fc42fc94152bd1d6b4cf3c513b8 P f42427cf15e945249ac2061305a6f745: Deleting WAL directory at /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790713339161492-17621-0/minicluster-data/ts-2/wal/wals/8e1b1fc42fc94152bd1d6b4cf3c513b8
I20260929 20:23:24.815335 25863 ts_tablet_manager.cc:1939] T 8e1b1fc42fc94152bd1d6b4cf3c513b8 P f42427cf15e945249ac2061305a6f745: Deleting consensus metadata
I20260929 20:23:24.815768 25841 tablet_bootstrap.cc:492] T 205f72f6f84945a5b8315b8123fa79e0 P f42427cf15e945249ac2061305a6f745: No bootstrap required, opened a new log
I20260929 20:23:24.815933 25841 ts_tablet_manager.cc:1397] T 205f72f6f84945a5b8315b8123fa79e0 P f42427cf15e945249ac2061305a6f745: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260929 20:23:24.816452 25865 tablet_bootstrap.cc:492] T 205f72f6f84945a5b8315b8123fa79e0 P f3630496a5db48fda15d65cfcfcf175f: Bootstrap starting.
I20260929 20:23:24.816494 25234 catalog_manager.cc:4841] TS f42427cf15e945249ac2061305a6f745 (127.17.53.67:38515): tablet 8e1b1fc42fc94152bd1d6b4cf3c513b8 (table test-table [id=2c5fd3a3d24a4b2d96299f333ab04d4d]) successfully deleted
I20260929 20:23:24.816713 25841 raft_consensus.cc:348] T 205f72f6f84945a5b8315b8123fa79e0 P f42427cf15e945249ac2061305a6f745 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3630496a5db48fda15d65cfcfcf175f" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 41053 } } peers { permanent_uuid: "70e535e06c4b430c9ec90f98dbc4f69b" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 40107 } } peers { permanent_uuid: "f42427cf15e945249ac2061305a6f745" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 38515 } }
I20260929 20:23:24.816954 25841 raft_consensus.cc:374] T 205f72f6f84945a5b8315b8123fa79e0 P f42427cf15e945249ac2061305a6f745 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:23:24.817023 25841 raft_consensus.cc:729] T 205f72f6f84945a5b8315b8123fa79e0 P f42427cf15e945249ac2061305a6f745 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f42427cf15e945249ac2061305a6f745, State: Initialized, Role: FOLLOWER
I20260929 20:23:24.817245 25841 consensus_queue.cc:260] T 205f72f6f84945a5b8315b8123fa79e0 P f42427cf15e945249ac2061305a6f745 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3630496a5db48fda15d65cfcfcf175f" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 41053 } } peers { permanent_uuid: "70e535e06c4b430c9ec90f98dbc4f69b" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 40107 } } peers { permanent_uuid: "f42427cf15e945249ac2061305a6f745" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 38515 } }
I20260929 20:23:24.817781 25841 ts_tablet_manager.cc:1428] T 205f72f6f84945a5b8315b8123fa79e0 P f42427cf15e945249ac2061305a6f745: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260929 20:23:24.818073 25865 tablet_bootstrap.cc:654] T 205f72f6f84945a5b8315b8123fa79e0 P f3630496a5db48fda15d65cfcfcf175f: Neither blocks nor log segments found. Creating new log.
I20260929 20:23:24.820261 25842 tablet_bootstrap.cc:492] T 205f72f6f84945a5b8315b8123fa79e0 P 70e535e06c4b430c9ec90f98dbc4f69b: No bootstrap required, opened a new log
I20260929 20:23:24.820449 25842 ts_tablet_manager.cc:1397] T 205f72f6f84945a5b8315b8123fa79e0 P 70e535e06c4b430c9ec90f98dbc4f69b: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.004s
I20260929 20:23:24.821247 25865 tablet_bootstrap.cc:492] T 205f72f6f84945a5b8315b8123fa79e0 P f3630496a5db48fda15d65cfcfcf175f: No bootstrap required, opened a new log
I20260929 20:23:24.821382 25865 ts_tablet_manager.cc:1397] T 205f72f6f84945a5b8315b8123fa79e0 P f3630496a5db48fda15d65cfcfcf175f: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.001s
I20260929 20:23:24.821296 25842 raft_consensus.cc:348] T 205f72f6f84945a5b8315b8123fa79e0 P 70e535e06c4b430c9ec90f98dbc4f69b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3630496a5db48fda15d65cfcfcf175f" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 41053 } } peers { permanent_uuid: "70e535e06c4b430c9ec90f98dbc4f69b" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 40107 } } peers { permanent_uuid: "f42427cf15e945249ac2061305a6f745" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 38515 } }
I20260929 20:23:24.821568 25842 raft_consensus.cc:374] T 205f72f6f84945a5b8315b8123fa79e0 P 70e535e06c4b430c9ec90f98dbc4f69b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:23:24.821650 25842 raft_consensus.cc:729] T 205f72f6f84945a5b8315b8123fa79e0 P 70e535e06c4b430c9ec90f98dbc4f69b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 70e535e06c4b430c9ec90f98dbc4f69b, State: Initialized, Role: FOLLOWER
I20260929 20:23:24.821882 25842 consensus_queue.cc:260] T 205f72f6f84945a5b8315b8123fa79e0 P 70e535e06c4b430c9ec90f98dbc4f69b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3630496a5db48fda15d65cfcfcf175f" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 41053 } } peers { permanent_uuid: "70e535e06c4b430c9ec90f98dbc4f69b" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 40107 } } peers { permanent_uuid: "f42427cf15e945249ac2061305a6f745" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 38515 } }
I20260929 20:23:24.822113 25865 raft_consensus.cc:348] T 205f72f6f84945a5b8315b8123fa79e0 P f3630496a5db48fda15d65cfcfcf175f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3630496a5db48fda15d65cfcfcf175f" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 41053 } } peers { permanent_uuid: "70e535e06c4b430c9ec90f98dbc4f69b" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 40107 } } peers { permanent_uuid: "f42427cf15e945249ac2061305a6f745" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 38515 } }
I20260929 20:23:24.822329 25865 raft_consensus.cc:374] T 205f72f6f84945a5b8315b8123fa79e0 P f3630496a5db48fda15d65cfcfcf175f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:23:24.822415 25865 raft_consensus.cc:729] T 205f72f6f84945a5b8315b8123fa79e0 P f3630496a5db48fda15d65cfcfcf175f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3630496a5db48fda15d65cfcfcf175f, State: Initialized, Role: FOLLOWER
I20260929 20:23:24.822461 25842 ts_tablet_manager.cc:1428] T 205f72f6f84945a5b8315b8123fa79e0 P 70e535e06c4b430c9ec90f98dbc4f69b: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260929 20:23:24.822678 25865 consensus_queue.cc:260] T 205f72f6f84945a5b8315b8123fa79e0 P f3630496a5db48fda15d65cfcfcf175f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3630496a5db48fda15d65cfcfcf175f" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 41053 } } peers { permanent_uuid: "70e535e06c4b430c9ec90f98dbc4f69b" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 40107 } } peers { permanent_uuid: "f42427cf15e945249ac2061305a6f745" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 38515 } }
I20260929 20:23:24.823477 25865 ts_tablet_manager.cc:1428] T 205f72f6f84945a5b8315b8123fa79e0 P f3630496a5db48fda15d65cfcfcf175f: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260929 20:23:24.990448 25824 raft_consensus.cc:482] T 205f72f6f84945a5b8315b8123fa79e0 P f3630496a5db48fda15d65cfcfcf175f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 20:23:24.990733 25824 raft_consensus.cc:504] T 205f72f6f84945a5b8315b8123fa79e0 P f3630496a5db48fda15d65cfcfcf175f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3630496a5db48fda15d65cfcfcf175f" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 41053 } } peers { permanent_uuid: "70e535e06c4b430c9ec90f98dbc4f69b" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 40107 } } peers { permanent_uuid: "f42427cf15e945249ac2061305a6f745" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 38515 } }
I20260929 20:23:24.991611 25824 leader_election.cc:288] T 205f72f6f84945a5b8315b8123fa79e0 P f3630496a5db48fda15d65cfcfcf175f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 70e535e06c4b430c9ec90f98dbc4f69b (127.17.53.66:40107), f42427cf15e945249ac2061305a6f745 (127.17.53.67:38515)
I20260929 20:23:24.992175 25551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "205f72f6f84945a5b8315b8123fa79e0" candidate_uuid: "f3630496a5db48fda15d65cfcfcf175f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "70e535e06c4b430c9ec90f98dbc4f69b" is_pre_election: true
I20260929 20:23:24.992208 25685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "205f72f6f84945a5b8315b8123fa79e0" candidate_uuid: "f3630496a5db48fda15d65cfcfcf175f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f42427cf15e945249ac2061305a6f745" is_pre_election: true
I20260929 20:23:24.992444 25685 raft_consensus.cc:2454] T 205f72f6f84945a5b8315b8123fa79e0 P f42427cf15e945249ac2061305a6f745 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f3630496a5db48fda15d65cfcfcf175f in term 0.
I20260929 20:23:24.992453 25551 raft_consensus.cc:2454] T 205f72f6f84945a5b8315b8123fa79e0 P 70e535e06c4b430c9ec90f98dbc4f69b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f3630496a5db48fda15d65cfcfcf175f in term 0.
I20260929 20:23:24.992878 25337 leader_election.cc:302] T 205f72f6f84945a5b8315b8123fa79e0 P f3630496a5db48fda15d65cfcfcf175f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f3630496a5db48fda15d65cfcfcf175f, f42427cf15e945249ac2061305a6f745; no voters:
I20260929 20:23:24.993251 25824 raft_consensus.cc:2781] T 205f72f6f84945a5b8315b8123fa79e0 P f3630496a5db48fda15d65cfcfcf175f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 20:23:24.993388 25824 raft_consensus.cc:482] T 205f72f6f84945a5b8315b8123fa79e0 P f3630496a5db48fda15d65cfcfcf175f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 20:23:24.993486 25824 raft_consensus.cc:3037] T 205f72f6f84945a5b8315b8123fa79e0 P f3630496a5db48fda15d65cfcfcf175f [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:23:24.995827 25824 raft_consensus.cc:504] T 205f72f6f84945a5b8315b8123fa79e0 P f3630496a5db48fda15d65cfcfcf175f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3630496a5db48fda15d65cfcfcf175f" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 41053 } } peers { permanent_uuid: "70e535e06c4b430c9ec90f98dbc4f69b" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 40107 } } peers { permanent_uuid: "f42427cf15e945249ac2061305a6f745" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 38515 } }
I20260929 20:23:24.996387 25824 leader_election.cc:288] T 205f72f6f84945a5b8315b8123fa79e0 P f3630496a5db48fda15d65cfcfcf175f [CANDIDATE]: Term 1 election: Requested vote from peers 70e535e06c4b430c9ec90f98dbc4f69b (127.17.53.66:40107), f42427cf15e945249ac2061305a6f745 (127.17.53.67:38515)
I20260929 20:23:24.996907 25551 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "205f72f6f84945a5b8315b8123fa79e0" candidate_uuid: "f3630496a5db48fda15d65cfcfcf175f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "70e535e06c4b430c9ec90f98dbc4f69b"
I20260929 20:23:24.996971 25685 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "205f72f6f84945a5b8315b8123fa79e0" candidate_uuid: "f3630496a5db48fda15d65cfcfcf175f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f42427cf15e945249ac2061305a6f745"
I20260929 20:23:24.997139 25551 raft_consensus.cc:3037] T 205f72f6f84945a5b8315b8123fa79e0 P 70e535e06c4b430c9ec90f98dbc4f69b [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:23:24.997233 25685 raft_consensus.cc:3037] T 205f72f6f84945a5b8315b8123fa79e0 P f42427cf15e945249ac2061305a6f745 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:23:24.999321 25551 raft_consensus.cc:2454] T 205f72f6f84945a5b8315b8123fa79e0 P 70e535e06c4b430c9ec90f98dbc4f69b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f3630496a5db48fda15d65cfcfcf175f in term 1.
I20260929 20:23:24.999730 25338 leader_election.cc:302] T 205f72f6f84945a5b8315b8123fa79e0 P f3630496a5db48fda15d65cfcfcf175f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 70e535e06c4b430c9ec90f98dbc4f69b, f3630496a5db48fda15d65cfcfcf175f; no voters:
I20260929 20:23:25.000016 25824 raft_consensus.cc:2781] T 205f72f6f84945a5b8315b8123fa79e0 P f3630496a5db48fda15d65cfcfcf175f [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:23:25.000252 25685 raft_consensus.cc:2454] T 205f72f6f84945a5b8315b8123fa79e0 P f42427cf15e945249ac2061305a6f745 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f3630496a5db48fda15d65cfcfcf175f in term 1.
I20260929 20:23:25.000521 25824 raft_consensus.cc:686] T 205f72f6f84945a5b8315b8123fa79e0 P f3630496a5db48fda15d65cfcfcf175f [term 1 LEADER]: Becoming Leader. State: Replica: f3630496a5db48fda15d65cfcfcf175f, State: Running, Role: LEADER
I20260929 20:23:25.000885 25824 consensus_queue.cc:237] T 205f72f6f84945a5b8315b8123fa79e0 P f3630496a5db48fda15d65cfcfcf175f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3630496a5db48fda15d65cfcfcf175f" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 41053 } } peers { permanent_uuid: "70e535e06c4b430c9ec90f98dbc4f69b" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 40107 } } peers { permanent_uuid: "f42427cf15e945249ac2061305a6f745" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 38515 } }
I20260929 20:23:25.003670 25249 catalog_manager.cc:5495] T 205f72f6f84945a5b8315b8123fa79e0 P f3630496a5db48fda15d65cfcfcf175f reported cstate change: term changed from 0 to 1, leader changed from <none> to f3630496a5db48fda15d65cfcfcf175f (127.17.53.65). New cstate: current_term: 1 leader_uuid: "f3630496a5db48fda15d65cfcfcf175f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3630496a5db48fda15d65cfcfcf175f" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 41053 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "70e535e06c4b430c9ec90f98dbc4f69b" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 40107 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f42427cf15e945249ac2061305a6f745" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 38515 } health_report { overall_health: UNKNOWN } } }
I20260929 20:23:25.064606 25871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:25.058618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46682 (local address 127.17.53.65:41053)
0929 20:23:25.059273 (+ 655us) server_negotiation.cc:206] Beginning negotiation
0929 20:23:25.059281 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 20:23:25.059306 (+ 25us) server_negotiation.cc:407] Connection header received
0929 20:23:25.059375 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:23:25.059382 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:23:25.059469 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:23:25.059581 (+ 112us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 20:23:25.059992 (+ 411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:25.061070 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:25.062556 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:25.063244 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:25.063375 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:25.063813 (+ 438us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 20:23:25.063926 (+ 113us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 20:23:25.064031 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0929 20:23:25.064188 (+ 157us) server_negotiation.cc:299] Negotiation successful
0929 20:23:25.064265 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":498,"thread_start_us":343,"threads_started":1}
I20260929 20:23:25.074671 25685 raft_consensus.cc:1261] T 205f72f6f84945a5b8315b8123fa79e0 P f42427cf15e945249ac2061305a6f745 [term 1 FOLLOWER]: Refusing update from remote peer f3630496a5db48fda15d65cfcfcf175f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:23:25.074779 25551 raft_consensus.cc:1261] T 205f72f6f84945a5b8315b8123fa79e0 P 70e535e06c4b430c9ec90f98dbc4f69b [term 1 FOLLOWER]: Refusing update from remote peer f3630496a5db48fda15d65cfcfcf175f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:23:25.075289 25824 consensus_queue.cc:1035] T 205f72f6f84945a5b8315b8123fa79e0 P f3630496a5db48fda15d65cfcfcf175f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f42427cf15e945249ac2061305a6f745" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 38515 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:23:25.075846 25824 consensus_queue.cc:1035] T 205f72f6f84945a5b8315b8123fa79e0 P f3630496a5db48fda15d65cfcfcf175f [LEADER]: Connected to new peer: Peer: permanent_uuid: "70e535e06c4b430c9ec90f98dbc4f69b" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 40107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:23:25.147228 25250 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57874:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 20:23:25.147526 25250 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57874:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 20:23:25.150738 25250 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d27fa57b2ddf4b438a0d3fd3a5271ac6: Sending DeleteTablet for 3 replicas of tablet 205f72f6f84945a5b8315b8123fa79e0
I20260929 20:23:25.151638 25665 tablet_service.cc:1514] Processing DeleteTablet for tablet 205f72f6f84945a5b8315b8123fa79e0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:23:25 UTC) from {username='oryx', principal='oryx/127.17.53.126@KRBTEST.COM'} at 127.0.0.1:47892
I20260929 20:23:25.151639 25386 tablet_service.cc:1514] Processing DeleteTablet for tablet 205f72f6f84945a5b8315b8123fa79e0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:23:25 UTC) from {username='oryx', principal='oryx/127.17.53.126@KRBTEST.COM'} at 127.0.0.1:46666
I20260929 20:23:25.151926 25863 tablet_replica.cc:330] T 205f72f6f84945a5b8315b8123fa79e0 P f42427cf15e945249ac2061305a6f745: stopping tablet replica
I20260929 20:23:25.152019 25862 tablet_replica.cc:330] T 205f72f6f84945a5b8315b8123fa79e0 P f3630496a5db48fda15d65cfcfcf175f: stopping tablet replica
I20260929 20:23:25.152124 25863 raft_consensus.cc:2229] T 205f72f6f84945a5b8315b8123fa79e0 P f42427cf15e945249ac2061305a6f745 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 20:23:25.152335 25863 raft_consensus.cc:2258] T 205f72f6f84945a5b8315b8123fa79e0 P f42427cf15e945249ac2061305a6f745 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:23:25.152328 25862 raft_consensus.cc:2229] T 205f72f6f84945a5b8315b8123fa79e0 P f3630496a5db48fda15d65cfcfcf175f [term 1 LEADER]: Raft consensus shutting down.
I20260929 20:23:25.152680 25862 raft_consensus.cc:2258] T 205f72f6f84945a5b8315b8123fa79e0 P f3630496a5db48fda15d65cfcfcf175f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:23:25.153059 17621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu pid 25321
I20260929 20:23:25.154119 25862 ts_tablet_manager.cc:1905] T 205f72f6f84945a5b8315b8123fa79e0 P f3630496a5db48fda15d65cfcfcf175f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:23:25.156780 25527 tablet_service.cc:1514] Processing DeleteTablet for tablet 205f72f6f84945a5b8315b8123fa79e0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:23:25 UTC) from {username='oryx', principal='oryx/127.17.53.126@KRBTEST.COM'} at 127.0.0.1:45766
I20260929 20:23:25.157183 25864 tablet_replica.cc:330] T 205f72f6f84945a5b8315b8123fa79e0 P 70e535e06c4b430c9ec90f98dbc4f69b: stopping tablet replica
I20260929 20:23:25.157413 25864 raft_consensus.cc:2229] T 205f72f6f84945a5b8315b8123fa79e0 P 70e535e06c4b430c9ec90f98dbc4f69b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 20:23:25.157620 25864 raft_consensus.cc:2258] T 205f72f6f84945a5b8315b8123fa79e0 P 70e535e06c4b430c9ec90f98dbc4f69b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:23:25.158669 25864 ts_tablet_manager.cc:1905] T 205f72f6f84945a5b8315b8123fa79e0 P 70e535e06c4b430c9ec90f98dbc4f69b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:23:25.158726 25863 ts_tablet_manager.cc:1905] T 205f72f6f84945a5b8315b8123fa79e0 P f42427cf15e945249ac2061305a6f745: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:23:25.165469 25863 ts_tablet_manager.cc:1918] T 205f72f6f84945a5b8315b8123fa79e0 P f42427cf15e945249ac2061305a6f745: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 20:23:25.165565 25862 ts_tablet_manager.cc:1918] T 205f72f6f84945a5b8315b8123fa79e0 P f3630496a5db48fda15d65cfcfcf175f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 20:23:25.165719 25862 log.cc:1196] T 205f72f6f84945a5b8315b8123fa79e0 P f3630496a5db48fda15d65cfcfcf175f: Deleting WAL directory at /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790713339161492-17621-0/minicluster-data/ts-0/wal/wals/205f72f6f84945a5b8315b8123fa79e0
I20260929 20:23:25.165831 25863 log.cc:1196] T 205f72f6f84945a5b8315b8123fa79e0 P f42427cf15e945249ac2061305a6f745: Deleting WAL directory at /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790713339161492-17621-0/minicluster-data/ts-2/wal/wals/205f72f6f84945a5b8315b8123fa79e0
I20260929 20:23:25.166143 25862 ts_tablet_manager.cc:1939] T 205f72f6f84945a5b8315b8123fa79e0 P f3630496a5db48fda15d65cfcfcf175f: Deleting consensus metadata
I20260929 20:23:25.166312 25863 ts_tablet_manager.cc:1939] T 205f72f6f84945a5b8315b8123fa79e0 P f42427cf15e945249ac2061305a6f745: Deleting consensus metadata
I20260929 20:23:25.167336 25235 catalog_manager.cc:4841] TS f3630496a5db48fda15d65cfcfcf175f (127.17.53.65:41053): tablet 205f72f6f84945a5b8315b8123fa79e0 (table test-table [id=24feb2db33384ee186c4a7d6dcdccfda]) successfully deleted
I20260929 20:23:25.167339 25234 catalog_manager.cc:4841] TS f42427cf15e945249ac2061305a6f745 (127.17.53.67:38515): tablet 205f72f6f84945a5b8315b8123fa79e0 (table test-table [id=24feb2db33384ee186c4a7d6dcdccfda]) successfully deleted
I20260929 20:23:25.168200 25864 ts_tablet_manager.cc:1918] T 205f72f6f84945a5b8315b8123fa79e0 P 70e535e06c4b430c9ec90f98dbc4f69b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260929 20:23:25.168375 25864 log.cc:1196] T 205f72f6f84945a5b8315b8123fa79e0 P 70e535e06c4b430c9ec90f98dbc4f69b: Deleting WAL directory at /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790713339161492-17621-0/minicluster-data/ts-1/wal/wals/205f72f6f84945a5b8315b8123fa79e0
I20260929 20:23:25.168902 25864 ts_tablet_manager.cc:1939] T 205f72f6f84945a5b8315b8123fa79e0 P 70e535e06c4b430c9ec90f98dbc4f69b: Deleting consensus metadata
Sep 29 20:23:25 dist-test-slave-zbnc krb5kdc[25205](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713404, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.17.53.65@KRBTEST.COM
I20260929 20:23:25.169960 25235 catalog_manager.cc:4841] TS 70e535e06c4b430c9ec90f98dbc4f69b (127.17.53.66:40107): tablet 205f72f6f84945a5b8315b8123fa79e0 (table test-table [id=24feb2db33384ee186c4a7d6dcdccfda]) successfully deleted
I20260929 20:23:25.177809 25871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:25.154345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46698 (local address 127.17.53.65:41053)
0929 20:23:25.156055 (+ 1710us) server_negotiation.cc:206] Beginning negotiation
0929 20:23:25.156063 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 20:23:25.156082 (+ 19us) server_negotiation.cc:407] Connection header received
0929 20:23:25.156138 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:23:25.156143 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:23:25.156230 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:23:25.156339 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:23:25.158208 (+ 1869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:25.159267 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:25.161035 (+ 1768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:25.161684 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:25.161775 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:25.170818 (+ 9043us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:23:25.170838 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:23:25.170885 (+ 47us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:23:25.170927 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:23:25.174961 (+ 4034us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:23:25.175684 (+ 723us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:23:25.175695 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:23:25.175699 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:23:25.175858 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:23:25.176210 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:23:25.176215 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:23:25.176219 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:23:25.176727 (+ 508us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:23:25.177069 (+ 342us) server_negotiation.cc:1035] Waiting for connection context
0929 20:23:25.177231 (+ 162us) server_negotiation.cc:299] Negotiation successful
0929 20:23:25.177446 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1564}
I20260929 20:23:25.178481 25452 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:23:25.305018 17621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu with pid 25321
I20260929 20:23:25.328253 17621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu pid 25461
Sep 29 20:23:25 dist-test-slave-zbnc krb5kdc[25205](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713404, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.17.53.66@KRBTEST.COM
I20260929 20:23:25.343664 25880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:25.329003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45794 (local address 127.17.53.66:40107)
0929 20:23:25.329458 (+ 455us) server_negotiation.cc:206] Beginning negotiation
0929 20:23:25.329464 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:23:25.329486 (+ 22us) server_negotiation.cc:407] Connection header received
0929 20:23:25.329538 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:23:25.329541 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:23:25.329596 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:23:25.329717 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:23:25.330950 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:25.331791 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:25.333030 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:25.333435 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:25.333571 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:25.337791 (+ 4220us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:23:25.337806 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:23:25.337823 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:23:25.337860 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:23:25.340778 (+ 2918us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:23:25.341463 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:23:25.341471 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:23:25.341473 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:23:25.341604 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:23:25.342037 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:23:25.342042 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:23:25.342045 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:23:25.342414 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:23:25.342733 (+ 319us) server_negotiation.cc:1035] Waiting for connection context
0929 20:23:25.343091 (+ 358us) server_negotiation.cc:299] Negotiation successful
0929 20:23:25.343307 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":340,"thread_start_us":224,"threads_started":1}
I20260929 20:23:25.344314 25592 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:23:25.488466 17621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu with pid 25461
I20260929 20:23:25.516760 17621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu pid 25601
Sep 29 20:23:25 dist-test-slave-zbnc krb5kdc[25205](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713404, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.17.53.67@KRBTEST.COM
I20260929 20:23:25.532460 25882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:25.517487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47906 (local address 127.17.53.67:38515)
0929 20:23:25.517992 (+ 505us) server_negotiation.cc:206] Beginning negotiation
0929 20:23:25.517996 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:23:25.518011 (+ 15us) server_negotiation.cc:407] Connection header received
0929 20:23:25.518069 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:23:25.518073 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:23:25.518159 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:23:25.518292 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:23:25.519595 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:25.520354 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:25.521586 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:25.521976 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:25.522117 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:25.526484 (+ 4367us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:23:25.526500 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:23:25.526517 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:23:25.526565 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:23:25.529639 (+ 3074us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:23:25.530365 (+ 726us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:23:25.530372 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:23:25.530375 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:23:25.530505 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:23:25.530966 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:23:25.530970 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:23:25.530972 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:23:25.531319 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:23:25.531617 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0929 20:23:25.531891 (+ 274us) server_negotiation.cc:299] Negotiation successful
0929 20:23:25.532108 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":356,"thread_start_us":235,"threads_started":1}
I20260929 20:23:25.533139 25730 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:23:25.656068 17621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu with pid 25601
I20260929 20:23:25.677685 17621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu pid 25218
I20260929 20:23:25.678897 25290 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:23:25.792459 17621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu with pid 25218
2026-09-29T20:23:25Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (7411 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790713339161492-17621-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 20:23:25 dist-test-slave-zbnc krb5kdc[25887](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 20:23:25 dist-test-slave-zbnc krb5kdc[25887](info): set up 2 sockets
Sep 29 20:23:25 dist-test-slave-zbnc krb5kdc[25887](info): commencing operation
krb5kdc: starting...
W20260929 20:23:26.973249 17621 mini_kdc.cc:121] Time spent starting KDC: real 1.136s user 0.003s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790713339161492-17621-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790713339161492-17621-0/krb5kdc/test-user.keytab.
Sep 29 20:23:27 dist-test-slave-zbnc krb5kdc[25887](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713407, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T20:23:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T20:23:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.17.53.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.53.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.53.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.126.keytab.
Entry for principal kudu/127.17.53.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.126.keytab.
I20260929 20:23:27.200884 17621 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
/tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790713339161492-17621-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790713339161492-17621-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790713339161492-17621-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790713339161492-17621-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.53.126:42635
--webserver_interface=127.17.53.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.126.keytab
--principal=kudu/127.17.53.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.53.84:42837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.53.126:42635
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790713339161492-17621-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790713339161492-17621-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790713339161492-17621-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:23:27.531981 25900 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:23:27.532315 25900 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:23:27.532474 25900 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:23:27.539633 25900 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:23:27.539798 25900 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:23:27.539909 25900 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:23:27.540011 25900 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:23:27.540083 25900 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:23:27.549863 25900 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.53.84:42837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790713339161492-17621-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790713339161492-17621-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.53.126:42635
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.126.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.53.126:42635
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790713339161492-17621-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.53.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790713339161492-17621-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:23:27.551268 25900 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:23:27.552603 25900 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:23:27.564795 25905 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:23:27.564879 25908 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:23:27.564993 25906 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:23:27.565310 25900 server_base.cc:1034] running on GCE node
I20260929 20:23:27.566244 25900 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:23:27.567790 25900 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:23:27.569039 25900 hybrid_clock.cc:648] HybridClock initialized: now 1790713407568972 us; error 58 us; skew 500 ppm
I20260929 20:23:27.569407 25900 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-29T20:23:27Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1844 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713339161492-17621-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 20:23:27 dist-test-slave-zbnc krb5kdc[25914](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 20:23:27 dist-test-slave-zbnc krb5kdc[25914](info): set up 2 sockets
Sep 29 20:23:27 dist-test-slave-zbnc krb5kdc[25914](info): commencing operation
krb5kdc: starting...
W20260929 20:23:28.757480 17621 mini_kdc.cc:121] Time spent starting KDC: real 1.075s user 0.006s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713339161492-17621-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713339161492-17621-0/krb5kdc/test-user.keytab.
Sep 29 20:23:28 dist-test-slave-zbnc krb5kdc[25914](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713408, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T20:23:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T20:23:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.17.53.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.53.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.53.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.126.keytab.
Entry for principal kudu/127.17.53.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.126.keytab.
I20260929 20:23:28.978564 17621 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
/tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713339161492-17621-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713339161492-17621-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713339161492-17621-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713339161492-17621-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.53.126:44263
--webserver_interface=127.17.53.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.126.keytab
--principal=kudu/127.17.53.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.53.84:38471
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.53.126:44263
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713339161492-17621-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713339161492-17621-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713339161492-17621-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:23:29.311602 25927 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:23:29.311901 25927 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:23:29.312068 25927 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:23:29.319375 25927 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:23:29.319482 25927 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:23:29.319586 25927 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:23:29.319662 25927 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:23:29.319718 25927 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:23:29.329604 25927 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.53.84:38471
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713339161492-17621-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713339161492-17621-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.53.126:44263
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.126.keytab
--principal=kudu/127.17.53.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.53.126:44263
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713339161492-17621-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.53.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713339161492-17621-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:23:29.331023 25927 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:23:29.332383 25927 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:23:29.344606 25932 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:23:29.344633 25933 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:23:29.345299 25927 server_base.cc:1034] running on GCE node
W20260929 20:23:29.345407 25935 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:23:29.346190 25927 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:23:29.347709 25927 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:23:29.348935 25927 hybrid_clock.cc:648] HybridClock initialized: now 1790713409348865 us; error 64 us; skew 500 ppm
I20260929 20:23:29.349291 25927 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:23:29 dist-test-slave-zbnc krb5kdc[25914](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713409, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.53.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:23:29.357762 25927 init.cc:377] Logged in from keytab as kudu/127.17.53.126@KRBTEST.COM (short username kudu)
I20260929 20:23:29.359491 25927 webserver.cc:460] Webserver started at http://127.17.53.126:44565/ using document root <none> and password file <none>
I20260929 20:23:29.360229 25927 fs_manager.cc:359] Metadata directory not provided
I20260929 20:23:29.360368 25927 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:23:29.360677 25927 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:23:29.364239 25927 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713339161492-17621-0/minicluster-data/master-0/data/instance:
uuid: "a6759ef7dbe34a85b7286192a7f8d4a6"
format_stamp: "Formatted at 2026-09-29 20:23:29 on dist-test-slave-zbnc"
server_key: "31e857db314540dceacb23b3b36830df"
server_key_iv: "83f35215041935e6bedfa9b1cfd81320"
server_key_version: "encryptionkey@0"
I20260929 20:23:29.365105 25927 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713339161492-17621-0/minicluster-data/master-0/wal/instance:
uuid: "a6759ef7dbe34a85b7286192a7f8d4a6"
format_stamp: "Formatted at 2026-09-29 20:23:29 on dist-test-slave-zbnc"
server_key: "31e857db314540dceacb23b3b36830df"
server_key_iv: "83f35215041935e6bedfa9b1cfd81320"
server_key_version: "encryptionkey@0"
I20260929 20:23:29.370764 25927 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260929 20:23:29.374150 25942 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:23:29.375453 25927 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260929 20:23:29.375633 25927 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713339161492-17621-0/minicluster-data/master-0/data,/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713339161492-17621-0/minicluster-data/master-0/wal
uuid: "a6759ef7dbe34a85b7286192a7f8d4a6"
format_stamp: "Formatted at 2026-09-29 20:23:29 on dist-test-slave-zbnc"
server_key: "31e857db314540dceacb23b3b36830df"
server_key_iv: "83f35215041935e6bedfa9b1cfd81320"
server_key_version: "encryptionkey@0"
I20260929 20:23:29.375833 25927 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713339161492-17621-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713339161492-17621-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713339161492-17621-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:23:29.389281 25927 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:23:29.393241 25927 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:23:29.393517 25927 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:23:29.411785 25927 rpc_server.cc:307] RPC server started. Bound to: 127.17.53.126:44263
I20260929 20:23:29.411790 25993 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.53.126:44263 every 8 connection(s)
I20260929 20:23:29.413256 25927 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713339161492-17621-0/minicluster-data/master-0/data/info.pb
I20260929 20:23:29.417173 25994 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:23:29.419409 17621 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu as pid 25927
I20260929 20:23:29.419638 17621 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713339161492-17621-0/minicluster-data/master-0/wal/instance
I20260929 20:23:29.420454 17621 external_mini_cluster.cc:1428] Setting key 1bc27df11b6f6af6c0e1099999421af5
I20260929 20:23:29.429131 25994 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a6759ef7dbe34a85b7286192a7f8d4a6: Bootstrap starting.
I20260929 20:23:29.432324 25994 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a6759ef7dbe34a85b7286192a7f8d4a6: Neither blocks nor log segments found. Creating new log.
I20260929 20:23:29.433323 25994 log.cc:824] T 00000000000000000000000000000000 P a6759ef7dbe34a85b7286192a7f8d4a6: Log is configured to *not* fsync() on all Append() calls
Sep 29 20:23:29 dist-test-slave-zbnc krb5kdc[25914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713408, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.53.126@KRBTEST.COM
I20260929 20:23:29.436281 25994 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a6759ef7dbe34a85b7286192a7f8d4a6: No bootstrap required, opened a new log
I20260929 20:23:29.442377 25997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:29.422588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52784 (local address 127.17.53.126:44263)
0929 20:23:29.423372 (+ 784us) server_negotiation.cc:206] Beginning negotiation
0929 20:23:29.423384 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 20:23:29.423427 (+ 43us) server_negotiation.cc:407] Connection header received
0929 20:23:29.425073 (+ 1646us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:23:29.425095 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:23:29.425491 (+ 396us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:23:29.425875 (+ 384us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:23:29.427439 (+ 1564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:29.428824 (+ 1385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:29.430090 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:29.430771 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:29.430992 (+ 221us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:29.435230 (+ 4238us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:23:29.435238 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:23:29.435252 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:23:29.435286 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:23:29.438320 (+ 3034us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:23:29.439066 (+ 746us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:23:29.439074 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:23:29.439087 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:23:29.439254 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:23:29.439628 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:23:29.439632 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:23:29.439637 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:23:29.440148 (+ 511us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:23:29.440511 (+ 363us) server_negotiation.cc:1035] Waiting for connection context
0929 20:23:29.440914 (+ 403us) server_negotiation.cc:299] Negotiation successful
0929 20:23:29.441337 (+ 423us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":571,"thread_start_us":353,"threads_started":1}
I20260929 20:23:29.444751 25994 raft_consensus.cc:348] T 00000000000000000000000000000000 P a6759ef7dbe34a85b7286192a7f8d4a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6759ef7dbe34a85b7286192a7f8d4a6" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 44263 } }
I20260929 20:23:29.445170 25994 raft_consensus.cc:374] T 00000000000000000000000000000000 P a6759ef7dbe34a85b7286192a7f8d4a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:23:29.445283 25994 raft_consensus.cc:729] T 00000000000000000000000000000000 P a6759ef7dbe34a85b7286192a7f8d4a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6759ef7dbe34a85b7286192a7f8d4a6, State: Initialized, Role: FOLLOWER
I20260929 20:23:29.445883 25994 consensus_queue.cc:260] T 00000000000000000000000000000000 P a6759ef7dbe34a85b7286192a7f8d4a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6759ef7dbe34a85b7286192a7f8d4a6" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 44263 } }
I20260929 20:23:29.446094 25994 raft_consensus.cc:388] T 00000000000000000000000000000000 P a6759ef7dbe34a85b7286192a7f8d4a6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 20:23:29.446185 25994 raft_consensus.cc:482] T 00000000000000000000000000000000 P a6759ef7dbe34a85b7286192a7f8d4a6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 20:23:29.446307 25994 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a6759ef7dbe34a85b7286192a7f8d4a6 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:23:29.448464 25994 raft_consensus.cc:504] T 00000000000000000000000000000000 P a6759ef7dbe34a85b7286192a7f8d4a6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6759ef7dbe34a85b7286192a7f8d4a6" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 44263 } }
I20260929 20:23:29.449007 25994 leader_election.cc:302] T 00000000000000000000000000000000 P a6759ef7dbe34a85b7286192a7f8d4a6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a6759ef7dbe34a85b7286192a7f8d4a6; no voters:
I20260929 20:23:29.449575 25994 leader_election.cc:288] T 00000000000000000000000000000000 P a6759ef7dbe34a85b7286192a7f8d4a6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 20:23:29.449720 25999 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a6759ef7dbe34a85b7286192a7f8d4a6 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:23:29.451067 25999 raft_consensus.cc:686] T 00000000000000000000000000000000 P a6759ef7dbe34a85b7286192a7f8d4a6 [term 1 LEADER]: Becoming Leader. State: Replica: a6759ef7dbe34a85b7286192a7f8d4a6, State: Running, Role: LEADER
I20260929 20:23:29.451556 25999 consensus_queue.cc:237] T 00000000000000000000000000000000 P a6759ef7dbe34a85b7286192a7f8d4a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6759ef7dbe34a85b7286192a7f8d4a6" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 44263 } }
I20260929 20:23:29.451706 25994 sys_catalog.cc:564] T 00000000000000000000000000000000 P a6759ef7dbe34a85b7286192a7f8d4a6 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:23:29.454825 26001 sys_catalog.cc:455] T 00000000000000000000000000000000 P a6759ef7dbe34a85b7286192a7f8d4a6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a6759ef7dbe34a85b7286192a7f8d4a6. Latest consensus state: current_term: 1 leader_uuid: "a6759ef7dbe34a85b7286192a7f8d4a6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6759ef7dbe34a85b7286192a7f8d4a6" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 44263 } } }
I20260929 20:23:29.455288 26001 sys_catalog.cc:458] T 00000000000000000000000000000000 P a6759ef7dbe34a85b7286192a7f8d4a6 [sys.catalog]: This master's current role is: LEADER
I20260929 20:23:29.454836 26000 sys_catalog.cc:455] T 00000000000000000000000000000000 P a6759ef7dbe34a85b7286192a7f8d4a6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a6759ef7dbe34a85b7286192a7f8d4a6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6759ef7dbe34a85b7286192a7f8d4a6" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 44263 } } }
I20260929 20:23:29.455444 26000 sys_catalog.cc:458] T 00000000000000000000000000000000 P a6759ef7dbe34a85b7286192a7f8d4a6 [sys.catalog]: This master's current role is: LEADER
I20260929 20:23:29.456781 26007 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:23:29.462374 26007 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:23:29.473456 26007 catalog_manager.cc:1347] Generated new cluster ID: ee3068784f93411ca8e626b94450cf9f
I20260929 20:23:29.473594 26007 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:23:29.483105 26007 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 20:23:29.484987 26007 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:23:29.498571 26007 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a6759ef7dbe34a85b7286192a7f8d4a6: Generated new TSK 0
I20260929 20:23:29.499748 26007 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.17.53.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.53.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.53.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.65.keytab.
Entry for principal kudu/127.17.53.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.65.keytab.
I20260929 20:23:29.559042 17621 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
/tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713339161492-17621-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713339161492-17621-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713339161492-17621-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713339161492-17621-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.53.65:0
--local_ip_for_outbound_sockets=127.17.53.65
--webserver_interface=127.17.53.65
--webserver_port=0
--tserver_master_addrs=127.17.53.126:44263
--keytab_file=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.65.keytab
--principal=kudu/127.17.53.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.53.84:38471
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713339161492-17621-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713339161492-17621-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713339161492-17621-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:23:29.894068 26020 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:23:29.894368 26020 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:23:29.894526 26020 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:23:29.901988 26020 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:23:29.902101 26020 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:23:29.902288 26020 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.53.65
I20260929 20:23:29.912178 26020 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.53.84:38471
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713339161492-17621-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713339161492-17621-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.65.keytab
--principal=kudu/127.17.53.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.53.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713339161492-17621-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.17.53.65
--webserver_port=0
--tserver_master_addrs=127.17.53.126:44263
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.53.65
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713339161492-17621-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:23:29.913504 26020 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:23:29.914920 26020 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:23:29.927930 26026 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:23:29.928005 26025 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:23:29.927971 26028 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:23:29.928687 26020 server_base.cc:1034] running on GCE node
I20260929 20:23:29.929340 26020 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:23:29.930275 26020 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:23:29.931504 26020 hybrid_clock.cc:648] HybridClock initialized: now 1790713409931413 us; error 78 us; skew 500 ppm
I20260929 20:23:29.931818 26020 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:23:29 dist-test-slave-zbnc krb5kdc[25914](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713409, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.53.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:23:29.934978 26020 init.cc:377] Logged in from keytab as kudu/127.17.53.65@KRBTEST.COM (short username kudu)
I20260929 20:23:29.936430 26020 webserver.cc:460] Webserver started at http://127.17.53.65:45033/ using document root <none> and password file <none>
I20260929 20:23:29.937147 26020 fs_manager.cc:359] Metadata directory not provided
I20260929 20:23:29.937279 26020 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:23:29.937606 26020 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:23:29.940598 26020 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713339161492-17621-0/minicluster-data/ts-0/data/instance:
uuid: "4573c780c2694f54b440088e1fcb2be8"
format_stamp: "Formatted at 2026-09-29 20:23:29 on dist-test-slave-zbnc"
server_key: "0ea9f777ee8003a12bdd0fe125aa92a5"
server_key_iv: "f67c1db113a68b0a462a401fb7649c92"
server_key_version: "encryptionkey@0"
I20260929 20:23:29.941366 26020 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713339161492-17621-0/minicluster-data/ts-0/wal/instance:
uuid: "4573c780c2694f54b440088e1fcb2be8"
format_stamp: "Formatted at 2026-09-29 20:23:29 on dist-test-slave-zbnc"
server_key: "0ea9f777ee8003a12bdd0fe125aa92a5"
server_key_iv: "f67c1db113a68b0a462a401fb7649c92"
server_key_version: "encryptionkey@0"
I20260929 20:23:29.946247 26020 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260929 20:23:29.949719 26035 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:23:29.951123 26020 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260929 20:23:29.951287 26020 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713339161492-17621-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713339161492-17621-0/minicluster-data/ts-0/wal
uuid: "4573c780c2694f54b440088e1fcb2be8"
format_stamp: "Formatted at 2026-09-29 20:23:29 on dist-test-slave-zbnc"
server_key: "0ea9f777ee8003a12bdd0fe125aa92a5"
server_key_iv: "f67c1db113a68b0a462a401fb7649c92"
server_key_version: "encryptionkey@0"
I20260929 20:23:29.951508 26020 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713339161492-17621-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713339161492-17621-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713339161492-17621-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:23:29.977535 26020 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:23:29.982311 26020 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:23:29.982551 26020 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:23:29.983569 26020 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:23:29.984907 26020 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:23:29.985006 26020 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:23:29.985146 26020 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:23:29.985213 26020 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:23:30.018190 26020 rpc_server.cc:307] RPC server started. Bound to: 127.17.53.65:37931
I20260929 20:23:30.018275 26147 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.53.65:37931 every 8 connection(s)
I20260929 20:23:30.019538 26020 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713339161492-17621-0/minicluster-data/ts-0/data/info.pb
I20260929 20:23:30.021041 17621 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu as pid 26020
I20260929 20:23:30.021265 17621 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713339161492-17621-0/minicluster-data/ts-0/wal/instance
I20260929 20:23:30.021912 17621 external_mini_cluster.cc:1428] Setting key 2483dd5dc4aa298b01f725cb0f80b88f
Sep 29 20:23:30 dist-test-slave-zbnc krb5kdc[25914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713409, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.53.65@KRBTEST.COM for kudu/127.17.53.126@KRBTEST.COM
I20260929 20:23:30.041074 26150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:30.022133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.53.65:43453 (local address 127.17.53.126:44263)
0929 20:23:30.022674 (+ 541us) server_negotiation.cc:206] Beginning negotiation
0929 20:23:30.022679 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:23:30.023886 (+ 1207us) server_negotiation.cc:407] Connection header received
0929 20:23:30.025772 (+ 1886us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:23:30.025779 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:23:30.025868 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:23:30.025982 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:23:30.027477 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:30.028669 (+ 1192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:30.031171 (+ 2502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:30.031576 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:30.031724 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:30.035172 (+ 3448us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:23:30.035177 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:23:30.035181 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:23:30.035213 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:23:30.037674 (+ 2461us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:23:30.038390 (+ 716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:23:30.038400 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:23:30.038402 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:23:30.038539 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:23:30.039025 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:23:30.039033 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:23:30.039038 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:23:30.039440 (+ 402us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:23:30.039593 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0929 20:23:30.040301 (+ 708us) server_negotiation.cc:299] Negotiation successful
0929 20:23:30.040575 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":394,"thread_start_us":250,"threads_started":1}
I20260929 20:23:30.041743 26151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:30.022843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:44263 (local address 127.17.53.65:43453)
0929 20:23:30.023676 (+ 833us) negotiation.cc:107] Waiting for socket to connect
0929 20:23:30.023730 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0929 20:23:30.025498 (+ 1768us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:23:30.026220 (+ 722us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:23:30.026228 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:23:30.026721 (+ 493us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:23:30.027316 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:23:30.027334 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:23:30.028882 (+ 1548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:23:30.028890 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:23:30.031028 (+ 2138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:23:30.031045 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:23:30.031812 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:23:30.031821 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:23:30.032063 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:30.032749 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:23:30.032771 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:23:30.034950 (+ 2179us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:23:30.037807 (+ 2857us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:23:30.037816 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:23:30.037833 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:23:30.038267 (+ 434us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:23:30.038654 (+ 387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:23:30.038659 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:23:30.038663 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:23:30.038848 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:23:30.039574 (+ 726us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:23:30.039580 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:23:30.039905 (+ 325us) client_negotiation.cc:769] Sending connection context
0929 20:23:30.040133 (+ 228us) client_negotiation.cc:240] Negotiation successful
0929 20:23:30.040529 (+ 396us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":511,"thread_start_us":289,"threads_started":1}
WARNING: no policy specified for kudu/127.17.53.66@KRBTEST.COM; defaulting to no policy
I20260929 20:23:30.043334 26148 heartbeater.cc:347] Connected to a master server at 127.17.53.126:44263
I20260929 20:23:30.043602 26148 heartbeater.cc:464] Registering TS with master...
I20260929 20:23:30.044353 26148 heartbeater.cc:511] Master 127.17.53.126:44263 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.53.66@KRBTEST.COM" created.
I20260929 20:23:30.046339 25959 ts_manager.cc:194] Registered new tserver with Master: 4573c780c2694f54b440088e1fcb2be8 (127.17.53.65:37931)
I20260929 20:23:30.047801 25959 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.53.65@KRBTEST.COM'} at 127.17.53.65:43453
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.53.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.66.keytab.
Entry for principal kudu/127.17.53.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.66.keytab.
I20260929 20:23:30.072347 17621 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
/tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713339161492-17621-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713339161492-17621-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713339161492-17621-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713339161492-17621-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.53.66:0
--local_ip_for_outbound_sockets=127.17.53.66
--webserver_interface=127.17.53.66
--webserver_port=0
--tserver_master_addrs=127.17.53.126:44263
--keytab_file=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.66.keytab
--principal=kudu/127.17.53.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.53.84:38471
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713339161492-17621-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713339161492-17621-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713339161492-17621-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:23:30.399190 26154 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:23:30.399477 26154 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:23:30.399636 26154 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:23:30.406898 26154 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:23:30.407006 26154 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:23:30.407191 26154 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.53.66
I20260929 20:23:30.420737 26154 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.53.84:38471
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713339161492-17621-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713339161492-17621-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.66.keytab
--principal=kudu/127.17.53.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.53.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713339161492-17621-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.17.53.66
--webserver_port=0
--tserver_master_addrs=127.17.53.126:44263
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.53.66
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713339161492-17621-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:23:30.422314 26154 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:23:30.423871 26154 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:23:30.438762 26160 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:23:30.438791 26162 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:23:30.438834 26159 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:23:30.439579 26154 server_base.cc:1034] running on GCE node
I20260929 20:23:30.440178 26154 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:23:30.441148 26154 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:23:30.442366 26154 hybrid_clock.cc:648] HybridClock initialized: now 1790713410442317 us; error 47 us; skew 500 ppm
I20260929 20:23:30.442675 26154 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:23:30 dist-test-slave-zbnc krb5kdc[25914](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713410, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.53.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:23:30.446056 26154 init.cc:377] Logged in from keytab as kudu/127.17.53.66@KRBTEST.COM (short username kudu)
I20260929 20:23:30.447741 26154 webserver.cc:460] Webserver started at http://127.17.53.66:34221/ using document root <none> and password file <none>
I20260929 20:23:30.448472 26154 fs_manager.cc:359] Metadata directory not provided
I20260929 20:23:30.448633 26154 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:23:30.448971 26154 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:23:30.452175 26154 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713339161492-17621-0/minicluster-data/ts-1/data/instance:
uuid: "d0f366760dca46fba86c3b49d3e2c4f8"
format_stamp: "Formatted at 2026-09-29 20:23:30 on dist-test-slave-zbnc"
server_key: "ad1e2988f23caf1adf2b1b3fada3016d"
server_key_iv: "a71acef01be7f5fbb119324b582a1dc4"
server_key_version: "encryptionkey@0"
I20260929 20:23:30.453033 26154 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713339161492-17621-0/minicluster-data/ts-1/wal/instance:
uuid: "d0f366760dca46fba86c3b49d3e2c4f8"
format_stamp: "Formatted at 2026-09-29 20:23:30 on dist-test-slave-zbnc"
server_key: "ad1e2988f23caf1adf2b1b3fada3016d"
server_key_iv: "a71acef01be7f5fbb119324b582a1dc4"
server_key_version: "encryptionkey@0"
I20260929 20:23:30.458470 26154 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260929 20:23:30.462229 26169 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:23:30.463630 26154 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 20:23:30.463814 26154 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713339161492-17621-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713339161492-17621-0/minicluster-data/ts-1/wal
uuid: "d0f366760dca46fba86c3b49d3e2c4f8"
format_stamp: "Formatted at 2026-09-29 20:23:30 on dist-test-slave-zbnc"
server_key: "ad1e2988f23caf1adf2b1b3fada3016d"
server_key_iv: "a71acef01be7f5fbb119324b582a1dc4"
server_key_version: "encryptionkey@0"
I20260929 20:23:30.464021 26154 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713339161492-17621-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713339161492-17621-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713339161492-17621-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:23:30.488377 26154 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:23:30.493320 26154 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:23:30.493662 26154 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:23:30.494704 26154 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:23:30.496193 26154 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:23:30.496310 26154 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:23:30.496469 26154 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:23:30.496553 26154 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:23:30.531666 26154 rpc_server.cc:307] RPC server started. Bound to: 127.17.53.66:40543
I20260929 20:23:30.531708 26281 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.53.66:40543 every 8 connection(s)
I20260929 20:23:30.533073 26154 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713339161492-17621-0/minicluster-data/ts-1/data/info.pb
I20260929 20:23:30.533979 17621 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu as pid 26154
I20260929 20:23:30.534168 17621 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713339161492-17621-0/minicluster-data/ts-1/wal/instance
I20260929 20:23:30.534762 17621 external_mini_cluster.cc:1428] Setting key 873403a2d8168530f501311587892b47
Sep 29 20:23:30 dist-test-slave-zbnc krb5kdc[25914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713410, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.53.66@KRBTEST.COM for kudu/127.17.53.126@KRBTEST.COM
I20260929 20:23:30.557317 26150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:30.536105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.53.66:43935 (local address 127.17.53.126:44263)
0929 20:23:30.536314 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0929 20:23:30.536321 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:23:30.537882 (+ 1561us) server_negotiation.cc:407] Connection header received
0929 20:23:30.540803 (+ 2921us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:23:30.540811 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:23:30.540895 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:23:30.540989 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:23:30.543245 (+ 2256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:30.544113 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:30.545867 (+ 1754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:30.546658 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:30.546801 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:30.550148 (+ 3347us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:23:30.550158 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:23:30.550165 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:23:30.550210 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:23:30.553805 (+ 3595us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:23:30.554584 (+ 779us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:23:30.554593 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:23:30.554599 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:23:30.554761 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:23:30.555266 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:23:30.555274 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:23:30.555280 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:23:30.555727 (+ 447us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:23:30.555895 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0929 20:23:30.556647 (+ 752us) server_negotiation.cc:299] Negotiation successful
0929 20:23:30.556888 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260929 20:23:30.558103 26284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:30.536785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:44263 (local address 127.17.53.66:43935)
0929 20:23:30.537694 (+ 909us) negotiation.cc:107] Waiting for socket to connect
0929 20:23:30.537749 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0929 20:23:30.540501 (+ 2752us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:23:30.541292 (+ 791us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:23:30.541307 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:23:30.542086 (+ 779us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:23:30.542980 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:23:30.543009 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:23:30.544279 (+ 1270us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:23:30.544287 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:23:30.545715 (+ 1428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:23:30.545728 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:23:30.546901 (+ 1173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:23:30.546908 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:23:30.547112 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:30.547801 (+ 689us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:23:30.547824 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:23:30.549919 (+ 2095us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:23:30.553937 (+ 4018us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:23:30.553944 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:23:30.553958 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:23:30.554430 (+ 472us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:23:30.554920 (+ 490us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:23:30.554923 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:23:30.554925 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:23:30.555133 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:23:30.555845 (+ 712us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:23:30.555851 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:23:30.556168 (+ 317us) client_negotiation.cc:769] Sending connection context
0929 20:23:30.556493 (+ 325us) client_negotiation.cc:240] Negotiation successful
0929 20:23:30.556892 (+ 399us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":600,"thread_start_us":351,"threads_started":1}
WARNING: no policy specified for kudu/127.17.53.67@KRBTEST.COM; defaulting to no policy
I20260929 20:23:30.559697 26282 heartbeater.cc:347] Connected to a master server at 127.17.53.126:44263
I20260929 20:23:30.560052 26282 heartbeater.cc:464] Registering TS with master...
I20260929 20:23:30.560753 26282 heartbeater.cc:511] Master 127.17.53.126:44263 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.53.67@KRBTEST.COM" created.
I20260929 20:23:30.562261 25959 ts_manager.cc:194] Registered new tserver with Master: d0f366760dca46fba86c3b49d3e2c4f8 (127.17.53.66:40543)
I20260929 20:23:30.563138 25959 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.53.66@KRBTEST.COM'} at 127.17.53.66:43935
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.53.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.67.keytab.
Entry for principal kudu/127.17.53.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.67.keytab.
I20260929 20:23:30.586694 17621 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
/tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713339161492-17621-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713339161492-17621-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713339161492-17621-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713339161492-17621-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.53.67:0
--local_ip_for_outbound_sockets=127.17.53.67
--webserver_interface=127.17.53.67
--webserver_port=0
--tserver_master_addrs=127.17.53.126:44263
--keytab_file=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.67.keytab
--principal=kudu/127.17.53.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.53.84:38471
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713339161492-17621-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713339161492-17621-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713339161492-17621-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:23:30.916030 26287 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:23:30.916332 26287 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:23:30.916491 26287 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:23:30.923904 26287 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:23:30.924038 26287 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:23:30.924214 26287 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.53.67
I20260929 20:23:30.934171 26287 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.53.84:38471
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713339161492-17621-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713339161492-17621-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.67.keytab
--principal=kudu/127.17.53.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.53.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713339161492-17621-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.17.53.67
--webserver_port=0
--tserver_master_addrs=127.17.53.126:44263
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.53.67
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713339161492-17621-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:23:30.935518 26287 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:23:30.936834 26287 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:23:30.949901 26292 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:23:30.949925 26295 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:23:30.949862 26293 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:23:30.950703 26287 server_base.cc:1034] running on GCE node
I20260929 20:23:30.951323 26287 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:23:30.952302 26287 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:23:30.953519 26287 hybrid_clock.cc:648] HybridClock initialized: now 1790713410953421 us; error 93 us; skew 500 ppm
I20260929 20:23:30.953836 26287 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:23:30 dist-test-slave-zbnc krb5kdc[25914](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713410, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.53.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:23:30.957016 26287 init.cc:377] Logged in from keytab as kudu/127.17.53.67@KRBTEST.COM (short username kudu)
I20260929 20:23:30.958489 26287 webserver.cc:460] Webserver started at http://127.17.53.67:39139/ using document root <none> and password file <none>
I20260929 20:23:30.959267 26287 fs_manager.cc:359] Metadata directory not provided
I20260929 20:23:30.959407 26287 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:23:30.959815 26287 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:23:30.962718 26287 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713339161492-17621-0/minicluster-data/ts-2/data/instance:
uuid: "88a1dbe6fd30406c9fbf5d80ab57d78b"
format_stamp: "Formatted at 2026-09-29 20:23:30 on dist-test-slave-zbnc"
server_key: "2ff3a22e29c7787e4ed2cc75a2701968"
server_key_iv: "fdfd2a218371bad3088674b98417a4af"
server_key_version: "encryptionkey@0"
I20260929 20:23:30.963590 26287 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713339161492-17621-0/minicluster-data/ts-2/wal/instance:
uuid: "88a1dbe6fd30406c9fbf5d80ab57d78b"
format_stamp: "Formatted at 2026-09-29 20:23:30 on dist-test-slave-zbnc"
server_key: "2ff3a22e29c7787e4ed2cc75a2701968"
server_key_iv: "fdfd2a218371bad3088674b98417a4af"
server_key_version: "encryptionkey@0"
I20260929 20:23:30.968809 26287 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260929 20:23:30.972484 26302 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:23:30.973878 26287 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 20:23:30.974057 26287 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713339161492-17621-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713339161492-17621-0/minicluster-data/ts-2/wal
uuid: "88a1dbe6fd30406c9fbf5d80ab57d78b"
format_stamp: "Formatted at 2026-09-29 20:23:30 on dist-test-slave-zbnc"
server_key: "2ff3a22e29c7787e4ed2cc75a2701968"
server_key_iv: "fdfd2a218371bad3088674b98417a4af"
server_key_version: "encryptionkey@0"
I20260929 20:23:30.974278 26287 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713339161492-17621-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713339161492-17621-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713339161492-17621-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:23:31.000334 26287 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:23:31.004567 26287 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:23:31.004899 26287 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:23:31.005939 26287 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:23:31.007418 26287 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:23:31.007520 26287 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:23:31.007682 26287 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:23:31.007753 26287 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:23:31.041689 26287 rpc_server.cc:307] RPC server started. Bound to: 127.17.53.67:39981
I20260929 20:23:31.041774 26414 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.53.67:39981 every 8 connection(s)
I20260929 20:23:31.043036 26287 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713339161492-17621-0/minicluster-data/ts-2/data/info.pb
I20260929 20:23:31.047484 17621 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu as pid 26287
I20260929 20:23:31.047735 17621 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713339161492-17621-0/minicluster-data/ts-2/wal/instance
I20260929 20:23:31.048368 17621 external_mini_cluster.cc:1428] Setting key 05d9880403ed525464f8e65f885a3342
I20260929 20:23:31.050707 26148 heartbeater.cc:503] Master 127.17.53.126:44263 was elected leader, sending a full tablet report...
Sep 29 20:23:31 dist-test-slave-zbnc krb5kdc[25914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713410, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.53.67@KRBTEST.COM for kudu/127.17.53.126@KRBTEST.COM
I20260929 20:23:31.071980 26150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:31.045570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.53.67:48265 (local address 127.17.53.126:44263)
0929 20:23:31.045752 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0929 20:23:31.045757 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:23:31.046843 (+ 1086us) server_negotiation.cc:407] Connection header received
0929 20:23:31.049249 (+ 2406us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:23:31.049255 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:23:31.049364 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:23:31.049468 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:23:31.051644 (+ 2176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:31.054579 (+ 2935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:31.060609 (+ 6030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:31.061183 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:31.061330 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:31.066308 (+ 4978us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:23:31.066314 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:23:31.066324 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:23:31.066357 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:23:31.069257 (+ 2900us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:23:31.069944 (+ 687us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:23:31.069948 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:23:31.069950 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:23:31.070101 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:23:31.070471 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:23:31.070475 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:23:31.070476 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:23:31.070746 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:23:31.070895 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0929 20:23:31.071576 (+ 681us) server_negotiation.cc:299] Negotiation successful
0929 20:23:31.071731 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 20:23:31.073009 26417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:31.045975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:44263 (local address 127.17.53.67:48265)
0929 20:23:31.046672 (+ 697us) negotiation.cc:107] Waiting for socket to connect
0929 20:23:31.046718 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0929 20:23:31.048981 (+ 2263us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:23:31.049716 (+ 735us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:23:31.049725 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:23:31.050383 (+ 658us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:23:31.051347 (+ 964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:23:31.051376 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:23:31.058062 (+ 6686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:23:31.058068 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:23:31.060458 (+ 2390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:23:31.060474 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:23:31.061683 (+ 1209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:23:31.061693 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:23:31.062017 (+ 324us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:31.062719 (+ 702us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:23:31.062742 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:23:31.066088 (+ 3346us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:23:31.069377 (+ 3289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:23:31.069384 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:23:31.069398 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:23:31.069835 (+ 437us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:23:31.070197 (+ 362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:23:31.070201 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:23:31.070203 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:23:31.070373 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:23:31.070910 (+ 537us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:23:31.070920 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:23:31.071263 (+ 343us) client_negotiation.cc:769] Sending connection context
0929 20:23:31.071485 (+ 222us) client_negotiation.cc:240] Negotiation successful
0929 20:23:31.071896 (+ 411us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":461,"thread_start_us":262,"threads_started":1}
I20260929 20:23:31.074349 26415 heartbeater.cc:347] Connected to a master server at 127.17.53.126:44263
I20260929 20:23:31.074622 26415 heartbeater.cc:464] Registering TS with master...
I20260929 20:23:31.075349 26415 heartbeater.cc:511] Master 127.17.53.126:44263 requested a full tablet report, sending...
I20260929 20:23:31.076749 25958 ts_manager.cc:194] Registered new tserver with Master: 88a1dbe6fd30406c9fbf5d80ab57d78b (127.17.53.67:39981)
I20260929 20:23:31.077582 17621 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 20:23:31.077772 25958 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.53.67@KRBTEST.COM'} at 127.17.53.67:48265
WARNING: no policy specified for oryx/127.17.53.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.17.53.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.17.53.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713339161492-17621-0/krb5kdc/oryx_127.17.53.65.keytab.
Entry for principal oryx/127.17.53.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713339161492-17621-0/krb5kdc/oryx_127.17.53.65.keytab.
WARNING: no policy specified for oryx/127.17.53.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.17.53.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.17.53.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713339161492-17621-0/krb5kdc/oryx_127.17.53.66.keytab.
Entry for principal oryx/127.17.53.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713339161492-17621-0/krb5kdc/oryx_127.17.53.66.keytab.
WARNING: no policy specified for oryx/127.17.53.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.17.53.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.17.53.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713339161492-17621-0/krb5kdc/oryx_127.17.53.67.keytab.
Entry for principal oryx/127.17.53.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713339161492-17621-0/krb5kdc/oryx_127.17.53.67.keytab.
I20260929 20:23:31.217425 17621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu pid 26020
Sep 29 20:23:31 dist-test-slave-zbnc krb5kdc[25914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713408, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.53.65@KRBTEST.COM
I20260929 20:23:31.234319 26425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:31.218665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42560 (local address 127.17.53.65:37931)
0929 20:23:31.219493 (+ 828us) server_negotiation.cc:206] Beginning negotiation
0929 20:23:31.219500 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:23:31.219530 (+ 30us) server_negotiation.cc:407] Connection header received
0929 20:23:31.219597 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:23:31.219604 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:23:31.219848 (+ 244us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:23:31.219996 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:23:31.221318 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:31.222525 (+ 1207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:31.223864 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:31.224489 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:31.224594 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:31.228591 (+ 3997us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:23:31.228603 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:23:31.228618 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:23:31.228656 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:23:31.231727 (+ 3071us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:23:31.232444 (+ 717us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:23:31.232510 (+ 66us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:23:31.232513 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:23:31.232633 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:23:31.233012 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:23:31.233017 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:23:31.233019 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:23:31.233374 (+ 355us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:23:31.233596 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0929 20:23:31.233850 (+ 254us) server_negotiation.cc:299] Negotiation successful
0929 20:23:31.234035 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":625,"thread_start_us":453,"threads_started":1}
I20260929 20:23:31.235735 26143 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:23:31.338042 17621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu with pid 26020
I20260929 20:23:31.354610 17621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu pid 26154
Sep 29 20:23:31 dist-test-slave-zbnc krb5kdc[25914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713408, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.53.66@KRBTEST.COM
I20260929 20:23:31.371150 26427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:31.355579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48930 (local address 127.17.53.66:40543)
0929 20:23:31.356080 (+ 501us) server_negotiation.cc:206] Beginning negotiation
0929 20:23:31.356095 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 20:23:31.356119 (+ 24us) server_negotiation.cc:407] Connection header received
0929 20:23:31.356178 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:23:31.356184 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:23:31.356390 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:23:31.356661 (+ 271us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:23:31.357903 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:31.359086 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:31.360401 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:31.360957 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:31.361103 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:31.365468 (+ 4365us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:23:31.365479 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:23:31.365494 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:23:31.365534 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:23:31.368381 (+ 2847us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:23:31.369047 (+ 666us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:23:31.369056 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:23:31.369060 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:23:31.369185 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:23:31.369538 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:23:31.369544 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:23:31.369547 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:23:31.369879 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:23:31.370168 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0929 20:23:31.370473 (+ 305us) server_negotiation.cc:299] Negotiation successful
0929 20:23:31.370746 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":367,"thread_start_us":239,"threads_started":1}
I20260929 20:23:31.372411 26277 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:23:31.453874 17621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu with pid 26154
I20260929 20:23:31.471923 17621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu pid 26287
Sep 29 20:23:31 dist-test-slave-zbnc krb5kdc[25914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713408, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.53.67@KRBTEST.COM
I20260929 20:23:31.487784 26429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:31.472725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41252 (local address 127.17.53.67:39981)
0929 20:23:31.473217 (+ 492us) server_negotiation.cc:206] Beginning negotiation
0929 20:23:31.473225 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 20:23:31.473255 (+ 30us) server_negotiation.cc:407] Connection header received
0929 20:23:31.473312 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:23:31.473318 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:23:31.473532 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:23:31.473706 (+ 174us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:23:31.475101 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:31.476078 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:31.477288 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:31.477917 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:31.478062 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:31.482533 (+ 4471us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:23:31.482543 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:23:31.482578 (+ 35us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:23:31.482615 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:23:31.485274 (+ 2659us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:23:31.485885 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:23:31.485892 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:23:31.485894 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:23:31.486009 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:23:31.486304 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:23:31.486307 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:23:31.486309 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:23:31.486637 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:23:31.486968 (+ 331us) server_negotiation.cc:1035] Waiting for connection context
0929 20:23:31.487210 (+ 242us) server_negotiation.cc:299] Negotiation successful
0929 20:23:31.487456 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":337,"thread_start_us":207,"threads_started":1}
I20260929 20:23:31.489223 26410 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:23:31.570329 17621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu with pid 26287
I20260929 20:23:31.586398 17621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu pid 25927
I20260929 20:23:31.587616 25989 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:23:31.690407 17621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu with pid 25927
I20260929 20:23:31.711567 17621 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
/tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713339161492-17621-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713339161492-17621-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713339161492-17621-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713339161492-17621-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.53.126:44263
--webserver_interface=127.17.53.126
--webserver_port=44565
--keytab_file=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.126.keytab
--principal=kudu/127.17.53.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.53.84:38471
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.53.126:44263
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713339161492-17621-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713339161492-17621-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713339161492-17621-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:23:32.041102 26432 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:23:32.041477 26432 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:23:32.041640 26432 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:23:32.049213 26432 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:23:32.049371 26432 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:23:32.049489 26432 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:23:32.049587 26432 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:23:32.049674 26432 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:23:32.059873 26432 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.53.84:38471
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713339161492-17621-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713339161492-17621-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.53.126:44263
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.126.keytab
--principal=kudu/127.17.53.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.53.126:44263
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713339161492-17621-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.53.126
--webserver_port=44565
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713339161492-17621-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:23:32.061233 26432 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:23:32.063234 26432 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:23:32.076766 26437 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:23:32.076728 26438 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:23:32.077153 26440 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:23:32.077590 26432 server_base.cc:1034] running on GCE node
I20260929 20:23:32.078397 26432 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:23:32.080576 26432 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:23:32.081812 26432 hybrid_clock.cc:648] HybridClock initialized: now 1790713412081750 us; error 72 us; skew 500 ppm
I20260929 20:23:32.082242 26432 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:23:32 dist-test-slave-zbnc krb5kdc[25914](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713412, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.53.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:23:32.085421 26432 init.cc:377] Logged in from keytab as kudu/127.17.53.126@KRBTEST.COM (short username kudu)
I20260929 20:23:32.086894 26432 webserver.cc:460] Webserver started at http://127.17.53.126:44565/ using document root <none> and password file <none>
I20260929 20:23:32.087604 26432 fs_manager.cc:359] Metadata directory not provided
I20260929 20:23:32.087757 26432 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:23:32.093101 26432 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260929 20:23:32.095889 26447 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:23:32.097188 26432 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 20:23:32.097389 26432 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713339161492-17621-0/minicluster-data/master-0/data,/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713339161492-17621-0/minicluster-data/master-0/wal
uuid: "a6759ef7dbe34a85b7286192a7f8d4a6"
format_stamp: "Formatted at 2026-09-29 20:23:29 on dist-test-slave-zbnc"
server_key: "31e857db314540dceacb23b3b36830df"
server_key_iv: "83f35215041935e6bedfa9b1cfd81320"
server_key_version: "encryptionkey@0"
I20260929 20:23:32.098390 26432 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713339161492-17621-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713339161492-17621-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713339161492-17621-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:23:32.119575 26432 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:23:32.123633 26432 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:23:32.123909 26432 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:23:32.142668 26432 rpc_server.cc:307] RPC server started. Bound to: 127.17.53.126:44263
I20260929 20:23:32.142692 26498 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.53.126:44263 every 8 connection(s)
I20260929 20:23:32.144253 26432 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713339161492-17621-0/minicluster-data/master-0/data/info.pb
I20260929 20:23:32.151130 26499 sys_catalog.cc:263] Verifying existing consensus state
I20260929 20:23:32.152740 17621 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu as pid 26432
I20260929 20:23:32.153585 17621 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
/tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713339161492-17621-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713339161492-17621-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713339161492-17621-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713339161492-17621-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.53.65:37931
--local_ip_for_outbound_sockets=127.17.53.65
--tserver_master_addrs=127.17.53.126:44263
--webserver_port=45033
--webserver_interface=127.17.53.65
--keytab_file=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.65.keytab
--principal=kudu/127.17.53.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.53.84:38471
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.17.53.65
--keytab_file=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713339161492-17621-0/krb5kdc/oryx_127.17.53.65.keytab with env {KRB5CCNAME=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713339161492-17621-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713339161492-17621-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713339161492-17621-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 20:23:32.154577 26499 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a6759ef7dbe34a85b7286192a7f8d4a6: Bootstrap starting.
I20260929 20:23:32.180594 26499 log.cc:824] T 00000000000000000000000000000000 P a6759ef7dbe34a85b7286192a7f8d4a6: Log is configured to *not* fsync() on all Append() calls
I20260929 20:23:32.191988 26499 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a6759ef7dbe34a85b7286192a7f8d4a6: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 20:23:32.192636 26499 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a6759ef7dbe34a85b7286192a7f8d4a6: Bootstrap complete.
I20260929 20:23:32.206135 26499 raft_consensus.cc:348] T 00000000000000000000000000000000 P a6759ef7dbe34a85b7286192a7f8d4a6 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6759ef7dbe34a85b7286192a7f8d4a6" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 44263 } }
I20260929 20:23:32.207384 26499 raft_consensus.cc:729] T 00000000000000000000000000000000 P a6759ef7dbe34a85b7286192a7f8d4a6 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6759ef7dbe34a85b7286192a7f8d4a6, State: Initialized, Role: FOLLOWER
I20260929 20:23:32.208189 26499 consensus_queue.cc:260] T 00000000000000000000000000000000 P a6759ef7dbe34a85b7286192a7f8d4a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6759ef7dbe34a85b7286192a7f8d4a6" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 44263 } }
I20260929 20:23:32.208492 26499 raft_consensus.cc:388] T 00000000000000000000000000000000 P a6759ef7dbe34a85b7286192a7f8d4a6 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 20:23:32.208658 26499 raft_consensus.cc:482] T 00000000000000000000000000000000 P a6759ef7dbe34a85b7286192a7f8d4a6 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 20:23:32.208871 26499 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a6759ef7dbe34a85b7286192a7f8d4a6 [term 1 FOLLOWER]: Advancing to term 2
I20260929 20:23:32.214775 26499 raft_consensus.cc:504] T 00000000000000000000000000000000 P a6759ef7dbe34a85b7286192a7f8d4a6 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6759ef7dbe34a85b7286192a7f8d4a6" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 44263 } }
I20260929 20:23:32.215415 26499 leader_election.cc:302] T 00000000000000000000000000000000 P a6759ef7dbe34a85b7286192a7f8d4a6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a6759ef7dbe34a85b7286192a7f8d4a6; no voters:
I20260929 20:23:32.216053 26499 leader_election.cc:288] T 00000000000000000000000000000000 P a6759ef7dbe34a85b7286192a7f8d4a6 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260929 20:23:32.216215 26503 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a6759ef7dbe34a85b7286192a7f8d4a6 [term 2 FOLLOWER]: Leader election won for term 2
I20260929 20:23:32.217625 26503 raft_consensus.cc:686] T 00000000000000000000000000000000 P a6759ef7dbe34a85b7286192a7f8d4a6 [term 2 LEADER]: Becoming Leader. State: Replica: a6759ef7dbe34a85b7286192a7f8d4a6, State: Running, Role: LEADER
I20260929 20:23:32.218228 26499 sys_catalog.cc:564] T 00000000000000000000000000000000 P a6759ef7dbe34a85b7286192a7f8d4a6 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:23:32.218171 26503 consensus_queue.cc:237] T 00000000000000000000000000000000 P a6759ef7dbe34a85b7286192a7f8d4a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6759ef7dbe34a85b7286192a7f8d4a6" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 44263 } }
I20260929 20:23:32.220654 26505 sys_catalog.cc:455] T 00000000000000000000000000000000 P a6759ef7dbe34a85b7286192a7f8d4a6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a6759ef7dbe34a85b7286192a7f8d4a6. Latest consensus state: current_term: 2 leader_uuid: "a6759ef7dbe34a85b7286192a7f8d4a6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6759ef7dbe34a85b7286192a7f8d4a6" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 44263 } } }
I20260929 20:23:32.221020 26505 sys_catalog.cc:458] T 00000000000000000000000000000000 P a6759ef7dbe34a85b7286192a7f8d4a6 [sys.catalog]: This master's current role is: LEADER
I20260929 20:23:32.221534 26504 sys_catalog.cc:455] T 00000000000000000000000000000000 P a6759ef7dbe34a85b7286192a7f8d4a6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "a6759ef7dbe34a85b7286192a7f8d4a6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a6759ef7dbe34a85b7286192a7f8d4a6" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 44263 } } }
I20260929 20:23:32.221884 26504 sys_catalog.cc:458] T 00000000000000000000000000000000 P a6759ef7dbe34a85b7286192a7f8d4a6 [sys.catalog]: This master's current role is: LEADER
I20260929 20:23:32.232072 26509 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:23:32.237924 26509 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:23:32.239401 26509 catalog_manager.cc:1259] Loaded cluster ID: ee3068784f93411ca8e626b94450cf9f
I20260929 20:23:32.239567 26509 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:23:32.243613 26509 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:23:32.245083 26509 catalog_manager.cc:5879] T 00000000000000000000000000000000 P a6759ef7dbe34a85b7286192a7f8d4a6: Loaded TSK: 0
I20260929 20:23:32.246634 26509 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260929 20:23:32.519239 26501 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:23:32.519577 26501 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:23:32.519740 26501 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:23:32.527060 26501 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:23:32.527199 26501 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:23:32.527370 26501 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.53.65
I20260929 20:23:32.537577 26501 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.53.84:38471
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713339161492-17621-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713339161492-17621-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713339161492-17621-0/krb5kdc/oryx_127.17.53.65.keytab
--principal=oryx/127.17.53.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.53.65:37931
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713339161492-17621-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.17.53.65
--webserver_port=45033
--tserver_master_addrs=127.17.53.126:44263
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.53.65
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713339161492-17621-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:23:32.539057 26501 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:23:32.540428 26501 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:23:32.554199 26526 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:23:32.554586 26525 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:23:32.556313 26528 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:23:32.556893 26501 server_base.cc:1034] running on GCE node
I20260929 20:23:32.557567 26501 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:23:32.558645 26501 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:23:32.559878 26501 hybrid_clock.cc:648] HybridClock initialized: now 1790713412559792 us; error 77 us; skew 500 ppm
I20260929 20:23:32.560212 26501 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:23:32 dist-test-slave-zbnc krb5kdc[25914](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713412, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.53.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:23:32.563854 26501 init.cc:377] Logged in from keytab as oryx/127.17.53.65@KRBTEST.COM (short username oryx)
I20260929 20:23:32.565459 26501 webserver.cc:460] Webserver started at http://127.17.53.65:45033/ using document root <none> and password file <none>
I20260929 20:23:32.566265 26501 fs_manager.cc:359] Metadata directory not provided
I20260929 20:23:32.566427 26501 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:23:32.571897 26501 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260929 20:23:32.574753 26535 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:23:32.576179 26501 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 20:23:32.576357 26501 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713339161492-17621-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713339161492-17621-0/minicluster-data/ts-0/wal
uuid: "4573c780c2694f54b440088e1fcb2be8"
format_stamp: "Formatted at 2026-09-29 20:23:29 on dist-test-slave-zbnc"
server_key: "0ea9f777ee8003a12bdd0fe125aa92a5"
server_key_iv: "f67c1db113a68b0a462a401fb7649c92"
server_key_version: "encryptionkey@0"
I20260929 20:23:32.577328 26501 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713339161492-17621-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713339161492-17621-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713339161492-17621-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:23:32.595227 26501 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:23:32.599157 26501 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:23:32.599427 26501 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:23:32.600359 26501 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:23:32.601665 26501 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:23:32.601795 26501 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:23:32.601945 26501 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:23:32.602051 26501 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:23:32.636060 26501 rpc_server.cc:307] RPC server started. Bound to: 127.17.53.65:37931
I20260929 20:23:32.636148 26647 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.53.65:37931 every 8 connection(s)
I20260929 20:23:32.637573 26501 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713339161492-17621-0/minicluster-data/ts-0/data/info.pb
I20260929 20:23:32.640183 17621 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu as pid 26501
I20260929 20:23:32.640806 17621 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
/tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713339161492-17621-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713339161492-17621-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713339161492-17621-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713339161492-17621-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.53.66:40543
--local_ip_for_outbound_sockets=127.17.53.66
--tserver_master_addrs=127.17.53.126:44263
--webserver_port=34221
--webserver_interface=127.17.53.66
--keytab_file=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.66.keytab
--principal=kudu/127.17.53.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.53.84:38471
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.17.53.66
--keytab_file=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713339161492-17621-0/krb5kdc/oryx_127.17.53.66.keytab with env {KRB5CCNAME=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713339161492-17621-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713339161492-17621-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713339161492-17621-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 20:23:32 dist-test-slave-zbnc krb5kdc[25914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.53.65@KRBTEST.COM for oryx/127.17.53.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 20:23:32 dist-test-slave-zbnc krb5kdc[25914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.53.65@KRBTEST.COM for oryx/127.17.53.126@KRBTEST.COM, Server not found in Kerberos database
W20260929 20:23:32.974213 26652 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:23:32.974558 26652 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:23:32.974735 26652 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:23:32.982185 26652 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:23:32.982318 26652 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:23:32.982532 26652 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.53.66
I20260929 20:23:32.992307 26652 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.53.84:38471
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713339161492-17621-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713339161492-17621-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713339161492-17621-0/krb5kdc/oryx_127.17.53.66.keytab
--principal=oryx/127.17.53.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.53.66:40543
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713339161492-17621-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.17.53.66
--webserver_port=34221
--tserver_master_addrs=127.17.53.126:44263
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.53.66
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713339161492-17621-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:23:32.993701 26652 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:23:32.995038 26652 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:23:33.009362 26658 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:23:33.009687 26661 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:23:33.009994 26659 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:23:33.010494 26652 server_base.cc:1034] running on GCE node
I20260929 20:23:33.011595 26652 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:23:33.013278 26652 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:23:33.014575 26652 hybrid_clock.cc:648] HybridClock initialized: now 1790713413014466 us; error 79 us; skew 500 ppm
I20260929 20:23:33.015156 26652 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:23:33 dist-test-slave-zbnc krb5kdc[25914](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713413, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.53.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:23:33.020396 26652 init.cc:377] Logged in from keytab as oryx/127.17.53.66@KRBTEST.COM (short username oryx)
I20260929 20:23:33.022132 26652 webserver.cc:460] Webserver started at http://127.17.53.66:34221/ using document root <none> and password file <none>
I20260929 20:23:33.022833 26652 fs_manager.cc:359] Metadata directory not provided
I20260929 20:23:33.023052 26652 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:23:33.028434 26652 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.003s
I20260929 20:23:33.031571 26668 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:23:33.033365 26652 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260929 20:23:33.033555 26652 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713339161492-17621-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713339161492-17621-0/minicluster-data/ts-1/wal
uuid: "d0f366760dca46fba86c3b49d3e2c4f8"
format_stamp: "Formatted at 2026-09-29 20:23:30 on dist-test-slave-zbnc"
server_key: "ad1e2988f23caf1adf2b1b3fada3016d"
server_key_iv: "a71acef01be7f5fbb119324b582a1dc4"
server_key_version: "encryptionkey@0"
I20260929 20:23:33.034617 26652 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713339161492-17621-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713339161492-17621-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713339161492-17621-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:23:33.060847 26652 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:23:33.065325 26652 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:23:33.065601 26652 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:23:33.066669 26652 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:23:33.068204 26652 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:23:33.068375 26652 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:23:33.068501 26652 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:23:33.068588 26652 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:23:33.114099 26652 rpc_server.cc:307] RPC server started. Bound to: 127.17.53.66:40543
I20260929 20:23:33.114307 26780 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.53.66:40543 every 8 connection(s)
I20260929 20:23:33.115808 26652 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713339161492-17621-0/minicluster-data/ts-1/data/info.pb
I20260929 20:23:33.126207 17621 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu as pid 26652
I20260929 20:23:33.126902 17621 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
/tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713339161492-17621-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713339161492-17621-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713339161492-17621-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713339161492-17621-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.53.67:39981
--local_ip_for_outbound_sockets=127.17.53.67
--tserver_master_addrs=127.17.53.126:44263
--webserver_port=39139
--webserver_interface=127.17.53.67
--keytab_file=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.67.keytab
--principal=kudu/127.17.53.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.53.84:38471
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.17.53.67
--keytab_file=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713339161492-17621-0/krb5kdc/oryx_127.17.53.67.keytab with env {KRB5CCNAME=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713339161492-17621-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713339161492-17621-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713339161492-17621-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 20:23:33 dist-test-slave-zbnc krb5kdc[25914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.53.66@KRBTEST.COM for oryx/127.17.53.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 20:23:33 dist-test-slave-zbnc krb5kdc[25914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.53.66@KRBTEST.COM for oryx/127.17.53.126@KRBTEST.COM, Server not found in Kerberos database
W20260929 20:23:33.755980 26651 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.101s user 0.000s sys 0.013s
I20260929 20:23:33.758095 26651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:32.640950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:44263 (local address 127.17.53.65:60897)
0929 20:23:32.643497 (+ 2547us) negotiation.cc:107] Waiting for socket to connect
0929 20:23:32.643551 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0929 20:23:32.645340 (+ 1789us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:23:32.647174 (+ 1834us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:23:32.647188 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:23:32.648003 (+ 815us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:23:32.648777 (+ 774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:23:32.648803 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:23:32.650369 (+ 1566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:23:32.650377 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:23:32.652583 (+ 2206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:23:32.652601 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:23:32.653647 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:23:32.653657 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:23:32.653976 (+ 319us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:32.655099 (+ 1123us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:23:32.655138 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:23:33.756907 (+1101769us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.53.126:44263: Server oryx/127.17.53.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":2249,"thread_start_us":2066,"threads_started":1}
W20260929 20:23:33.758093 26650 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:23:32.640777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.53.65:60897 (local address 127.17.53.126:44263)
0929 20:23:32.641621 (+ 844us) server_negotiation.cc:206] Beginning negotiation
0929 20:23:32.641640 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0929 20:23:32.643714 (+ 2074us) server_negotiation.cc:407] Connection header received
0929 20:23:32.645958 (+ 2244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:23:32.645980 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:23:32.646534 (+ 554us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:23:32.646937 (+ 403us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:23:32.648977 (+ 2040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:32.650197 (+ 1220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:32.652727 (+ 2530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:32.653397 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:32.653537 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:33.757301 (+1103764us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.53.65:60897: BlockingRecv error: recv got EOF from 127.17.53.65:60897 (error 108)
Metrics: {"server-negotiator.queue_time_us":553,"thread_start_us":332,"threads_started":1}
W20260929 20:23:33.759737 26648 heartbeater.cc:644] Failed to heartbeat to 127.17.53.126:44263 (0 consecutive failures): Not authorized: Failed to ping master at 127.17.53.126:44263: Client connection negotiation failed: client connection to 127.17.53.126:44263: Server oryx/127.17.53.126@KRBTEST.COM not found in Kerberos database
Sep 29 20:23:33 dist-test-slave-zbnc krb5kdc[25914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.53.65@KRBTEST.COM for oryx/127.17.53.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 20:23:33 dist-test-slave-zbnc krb5kdc[25914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.53.65@KRBTEST.COM for oryx/127.17.53.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 20:23:33.769322 26651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:33.760495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:44263 (local address 127.17.53.65:55595)
0929 20:23:33.760862 (+ 367us) negotiation.cc:107] Waiting for socket to connect
0929 20:23:33.760876 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 20:23:33.761033 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:23:33.761363 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:23:33.761367 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:23:33.761731 (+ 364us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:23:33.761892 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:23:33.761901 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:23:33.762924 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:23:33.762929 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:23:33.763877 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:23:33.763888 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:23:33.764640 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:23:33.764646 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:23:33.764814 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:33.765437 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:23:33.765457 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:23:33.769015 (+ 3558us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.53.126:44263: Server oryx/127.17.53.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":274}
W20260929 20:23:33.769447 26650 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:23:33.760588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.53.65:55595 (local address 127.17.53.126:44263)
0929 20:23:33.760749 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0929 20:23:33.760754 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:23:33.760954 (+ 200us) server_negotiation.cc:407] Connection header received
0929 20:23:33.761160 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:23:33.761165 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:23:33.761235 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:23:33.761335 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:23:33.762035 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:33.762766 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:33.764033 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:33.764463 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:33.764588 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:33.769210 (+ 4622us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.53.65:55595: BlockingRecv error: recv got EOF from 127.17.53.65:55595 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 29 20:23:33 dist-test-slave-zbnc krb5kdc[25914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.53.65@KRBTEST.COM for oryx/127.17.53.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 20:23:33 dist-test-slave-zbnc krb5kdc[25914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.53.65@KRBTEST.COM for oryx/127.17.53.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 20:23:33.778389 26651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:33.770233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:44263 (local address 127.17.53.65:32895)
0929 20:23:33.770518 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0929 20:23:33.770531 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 20:23:33.770651 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:23:33.771009 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:23:33.771014 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:23:33.771305 (+ 291us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:23:33.771481 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:23:33.771490 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:23:33.772467 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:23:33.772471 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:23:33.773396 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:23:33.773407 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:23:33.774091 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:23:33.774097 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:23:33.774264 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:33.774851 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:23:33.774912 (+ 61us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:23:33.778104 (+ 3192us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.53.126:44263: Server oryx/127.17.53.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":203}
W20260929 20:23:33.778535 26650 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:23:33.770284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.53.65:32895 (local address 127.17.53.126:44263)
0929 20:23:33.770420 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0929 20:23:33.770424 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:23:33.770596 (+ 172us) server_negotiation.cc:407] Connection header received
0929 20:23:33.770764 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:23:33.770769 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:23:33.770847 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:23:33.770981 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:23:33.771608 (+ 627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:33.772316 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:33.773542 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:33.773942 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:33.774081 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:33.778300 (+ 4219us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.53.65:32895: BlockingRecv error: recv got EOF from 127.17.53.65:32895 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260929 20:23:33.778985 26648 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 20:23:33.825770 26785 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:23:33.826391 26785 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:23:33.826624 26785 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:23:33.845875 26785 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:23:33.846118 26785 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:23:33.846436 26785 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.53.67
I20260929 20:23:33.872100 26785 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.53.84:38471
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713339161492-17621-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713339161492-17621-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713339161492-17621-0/krb5kdc/oryx_127.17.53.67.keytab
--principal=oryx/127.17.53.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.53.67:39981
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713339161492-17621-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.17.53.67
--webserver_port=39139
--tserver_master_addrs=127.17.53.126:44263
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.53.67
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713339161492-17621-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:23:33.874676 26785 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:23:33.877251 26785 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:23:33.896736 26792 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:23:33.896736 26791 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:23:33.898211 26794 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:23:33.898357 26785 server_base.cc:1034] running on GCE node
I20260929 20:23:33.899492 26785 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:23:33.901185 26785 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:23:33.902513 26785 hybrid_clock.cc:648] HybridClock initialized: now 1790713413902425 us; error 59 us; skew 500 ppm
I20260929 20:23:33.903118 26785 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:23:33 dist-test-slave-zbnc krb5kdc[25914](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713413, etypes {rep=17 tkt=17 ses=17}, oryx/127.17.53.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:23:33.909055 26785 init.cc:377] Logged in from keytab as oryx/127.17.53.67@KRBTEST.COM (short username oryx)
I20260929 20:23:33.911727 26785 webserver.cc:460] Webserver started at http://127.17.53.67:39139/ using document root <none> and password file <none>
I20260929 20:23:33.913127 26785 fs_manager.cc:359] Metadata directory not provided
I20260929 20:23:33.913372 26785 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:23:33.923995 26785 fs_manager.cc:709] Time spent opening directory manager: real 0.007s user 0.007s sys 0.000s
I20260929 20:23:33.928769 26801 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:23:33.931203 26785 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.007s sys 0.000s
I20260929 20:23:33.931466 26785 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713339161492-17621-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713339161492-17621-0/minicluster-data/ts-2/wal
uuid: "88a1dbe6fd30406c9fbf5d80ab57d78b"
format_stamp: "Formatted at 2026-09-29 20:23:30 on dist-test-slave-zbnc"
server_key: "2ff3a22e29c7787e4ed2cc75a2701968"
server_key_iv: "fdfd2a218371bad3088674b98417a4af"
server_key_version: "encryptionkey@0"
I20260929 20:23:33.933593 26785 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713339161492-17621-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713339161492-17621-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713339161492-17621-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:23:33.984009 26785 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:23:33.988479 26785 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:23:33.988770 26785 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:23:33.989825 26785 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:23:33.991366 26785 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:23:33.991498 26785 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:23:33.991640 26785 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:23:33.991729 26785 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:23:34.036341 26785 rpc_server.cc:307] RPC server started. Bound to: 127.17.53.67:39981
I20260929 20:23:34.036391 26913 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.53.67:39981 every 8 connection(s)
I20260929 20:23:34.038033 26785 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790713339161492-17621-0/minicluster-data/ts-2/data/info.pb
I20260929 20:23:34.041590 17621 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu as pid 26785
Sep 29 20:23:34 dist-test-slave-zbnc krb5kdc[25914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.53.67@KRBTEST.COM for oryx/127.17.53.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 20:23:34 dist-test-slave-zbnc krb5kdc[25914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.53.67@KRBTEST.COM for oryx/127.17.53.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 20:23:34.079031 26918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:34.043300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36792 (local address 127.17.53.126:44263)
0929 20:23:34.051253 (+ 7953us) server_negotiation.cc:206] Beginning negotiation
0929 20:23:34.051257 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:23:34.056259 (+ 5002us) server_negotiation.cc:407] Connection header received
0929 20:23:34.056460 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:23:34.056466 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:23:34.056542 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:23:34.056647 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:23:34.058654 (+ 2007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:34.060505 (+ 1851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:34.062631 (+ 2126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:34.063680 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:34.063858 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:34.067713 (+ 3855us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:23:34.067729 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:23:34.067767 (+ 38us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:23:34.067845 (+ 78us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:23:34.073520 (+ 5675us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:23:34.074770 (+ 1250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:23:34.074785 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:23:34.074800 (+ 15us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:23:34.075152 (+ 352us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:23:34.075794 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:23:34.075802 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:23:34.075808 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:23:34.076718 (+ 910us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:23:34.077200 (+ 482us) server_negotiation.cc:1035] Waiting for connection context
0929 20:23:34.077776 (+ 576us) server_negotiation.cc:299] Negotiation successful
0929 20:23:34.078353 (+ 577us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7845,"thread_start_us":7712,"threads_started":1}
W20260929 20:23:34.405628 26784 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.272s user 0.009s sys 0.009s
W20260929 20:23:34.407238 26783 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:23:33.119717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.53.66:45551 (local address 127.17.53.126:44263)
0929 20:23:33.120370 (+ 653us) server_negotiation.cc:206] Beginning negotiation
0929 20:23:33.120380 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 20:23:33.121294 (+ 914us) server_negotiation.cc:407] Connection header received
0929 20:23:33.123347 (+ 2053us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:23:33.123356 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:23:33.123476 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:23:33.123650 (+ 174us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:23:33.125194 (+ 1544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:33.127000 (+ 1806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:33.130627 (+ 3627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:33.131418 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:33.131572 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:34.406823 (+1275251us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.53.66:45551: BlockingRecv error: recv got EOF from 127.17.53.66:45551 (error 108)
Metrics: {"server-negotiator.queue_time_us":461,"thread_start_us":291,"threads_started":1}
I20260929 20:23:34.407899 26784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:33.120303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:44263 (local address 127.17.53.66:45551)
0929 20:23:33.121124 (+ 821us) negotiation.cc:107] Waiting for socket to connect
0929 20:23:33.121170 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0929 20:23:33.123064 (+ 1894us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:23:33.123854 (+ 790us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:23:33.123862 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:23:33.124420 (+ 558us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:23:33.125010 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:23:33.125028 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:23:33.127297 (+ 2269us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:23:33.127307 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:23:33.130388 (+ 3081us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:23:33.130413 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:23:33.131779 (+ 1366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:23:33.131789 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:23:33.132268 (+ 479us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:33.133390 (+ 1122us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:23:33.133444 (+ 54us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:23:34.406605 (+1273161us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.53.126:44263: Server oryx/127.17.53.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":540,"thread_start_us":363,"threads_started":1}
W20260929 20:23:34.408886 26781 heartbeater.cc:644] Failed to heartbeat to 127.17.53.126:44263 (0 consecutive failures): Not authorized: Failed to ping master at 127.17.53.126:44263: Client connection negotiation failed: client connection to 127.17.53.126:44263: Server oryx/127.17.53.126@KRBTEST.COM not found in Kerberos database
Sep 29 20:23:34 dist-test-slave-zbnc krb5kdc[25914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.53.66@KRBTEST.COM for oryx/127.17.53.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 20:23:34 dist-test-slave-zbnc krb5kdc[25914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.53.66@KRBTEST.COM for oryx/127.17.53.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 20:23:34.421911 26784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:34.409664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:44263 (local address 127.17.53.66:37049)
0929 20:23:34.409962 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0929 20:23:34.409977 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 20:23:34.410122 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:23:34.410439 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:23:34.410444 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:23:34.410841 (+ 397us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:23:34.411059 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:23:34.411068 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:23:34.412064 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:23:34.412072 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:23:34.413024 (+ 952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:23:34.413035 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:23:34.413790 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:23:34.413796 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:23:34.413955 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:34.417510 (+ 3555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:23:34.417529 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:23:34.421593 (+ 4064us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.53.126:44263: Server oryx/127.17.53.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":199}
W20260929 20:23:34.422026 26783 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:23:34.409689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.53.66:37049 (local address 127.17.53.126:44263)
0929 20:23:34.409843 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0929 20:23:34.409847 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:23:34.410060 (+ 213us) server_negotiation.cc:407] Connection header received
0929 20:23:34.410241 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:23:34.410247 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:23:34.410315 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:23:34.410414 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:23:34.411189 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:34.411925 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:34.413183 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:34.413610 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:34.413738 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:34.421794 (+ 8056us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.53.66:37049: BlockingRecv error: recv got EOF from 127.17.53.66:37049 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Sep 29 20:23:34 dist-test-slave-zbnc krb5kdc[25914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.53.66@KRBTEST.COM for oryx/127.17.53.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 20:23:34 dist-test-slave-zbnc krb5kdc[25914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.53.66@KRBTEST.COM for oryx/127.17.53.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 20:23:34.432106 26784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:34.422618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:44263 (local address 127.17.53.66:38691)
0929 20:23:34.423070 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0929 20:23:34.423088 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 20:23:34.423194 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:23:34.423540 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:23:34.423545 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:23:34.423896 (+ 351us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:23:34.424058 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:23:34.424066 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:23:34.425205 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:23:34.425211 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:23:34.426160 (+ 949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:23:34.426170 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:23:34.426918 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:23:34.426926 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:23:34.427102 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:34.427718 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:23:34.427734 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:23:34.431814 (+ 4080us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.53.126:44263: Server oryx/127.17.53.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":350}
W20260929 20:23:34.432405 26781 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 20:23:34.432642 26783 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:23:34.422909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.53.66:38691 (local address 127.17.53.126:44263)
0929 20:23:34.423061 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0929 20:23:34.423067 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:23:34.423135 (+ 68us) server_negotiation.cc:407] Connection header received
0929 20:23:34.423325 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:23:34.423331 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:23:34.423411 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:23:34.423510 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:23:34.424199 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:34.425064 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:34.426292 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:34.426714 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:34.426839 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:34.432010 (+ 5171us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.53.66:38691: BlockingRecv error: recv got EOF from 127.17.53.66:38691 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
Sep 29 20:23:34 dist-test-slave-zbnc krb5kdc[25914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.53.65@KRBTEST.COM for oryx/127.17.53.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 20:23:34 dist-test-slave-zbnc krb5kdc[25914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.53.65@KRBTEST.COM for oryx/127.17.53.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 20:23:34.789834 26920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:34.779806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:44263 (local address 127.17.53.65:53169)
0929 20:23:34.780516 (+ 710us) negotiation.cc:107] Waiting for socket to connect
0929 20:23:34.780538 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 20:23:34.780699 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:23:34.781088 (+ 389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:23:34.781093 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:23:34.781641 (+ 548us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:23:34.781792 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:23:34.781801 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:23:34.783067 (+ 1266us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:23:34.783072 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:23:34.784488 (+ 1416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:23:34.784503 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:23:34.785221 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:23:34.785228 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:23:34.785404 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:34.786049 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:23:34.786067 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:23:34.789486 (+ 3419us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.53.126:44263: Server oryx/127.17.53.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":603,"thread_start_us":442,"threads_started":1}
W20260929 20:23:34.789920 26783 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:23:34.779832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.53.65:53169 (local address 127.17.53.126:44263)
0929 20:23:34.779997 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0929 20:23:34.780001 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:23:34.780626 (+ 625us) server_negotiation.cc:407] Connection header received
0929 20:23:34.780825 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:23:34.780831 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:23:34.780916 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:23:34.781048 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:23:34.781936 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:34.782905 (+ 969us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:34.784630 (+ 1725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:34.785046 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:34.785185 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:34.789699 (+ 4514us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.53.65:53169: BlockingRecv error: recv got EOF from 127.17.53.65:53169 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
W20260929 20:23:35.162786 26916 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.105s user 0.000s sys 0.015s
W20260929 20:23:35.164000 26650 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:23:34.041939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.53.67:41781 (local address 127.17.53.126:44263)
0929 20:23:34.042282 (+ 343us) server_negotiation.cc:206] Beginning negotiation
0929 20:23:34.042292 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 20:23:34.043510 (+ 1218us) server_negotiation.cc:407] Connection header received
0929 20:23:34.047017 (+ 3507us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:23:34.047025 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:23:34.047138 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:23:34.047234 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:23:34.050162 (+ 2928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:34.051667 (+ 1505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:34.054944 (+ 3277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:34.055949 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:34.056174 (+ 225us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:35.163752 (+1107578us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.53.67:41781: BlockingRecv error: recv got EOF from 127.17.53.67:41781 (error 108)
Metrics: {"server-negotiator.queue_time_us":139}
I20260929 20:23:35.164659 26916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:34.042509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:44263 (local address 127.17.53.67:41781)
0929 20:23:34.043346 (+ 837us) negotiation.cc:107] Waiting for socket to connect
0929 20:23:34.043398 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0929 20:23:34.046588 (+ 3190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:23:34.047911 (+ 1323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:23:34.047928 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:23:34.048915 (+ 987us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:23:34.049919 (+ 1004us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:23:34.049951 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:23:34.051948 (+ 1997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:23:34.051956 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:23:34.054685 (+ 2729us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:23:34.054706 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:23:34.056264 (+ 1558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:23:34.056273 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:23:34.056572 (+ 299us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:34.057589 (+ 1017us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:23:34.057624 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:23:35.163604 (+1105980us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.53.126:44263: Server oryx/127.17.53.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":531,"thread_start_us":797,"threads_started":1}
W20260929 20:23:35.165490 26914 heartbeater.cc:644] Failed to heartbeat to 127.17.53.126:44263 (0 consecutive failures): Not authorized: Failed to ping master at 127.17.53.126:44263: Client connection negotiation failed: client connection to 127.17.53.126:44263: Server oryx/127.17.53.126@KRBTEST.COM not found in Kerberos database
Sep 29 20:23:35 dist-test-slave-zbnc krb5kdc[25914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.53.67@KRBTEST.COM for oryx/127.17.53.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 20:23:35 dist-test-slave-zbnc krb5kdc[25914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.53.67@KRBTEST.COM for oryx/127.17.53.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 20:23:35.174639 26916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:35.166142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:44263 (local address 127.17.53.67:36917)
0929 20:23:35.166326 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0929 20:23:35.166348 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 20:23:35.166447 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:23:35.166721 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:23:35.166724 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:23:35.167115 (+ 391us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:23:35.167276 (+ 161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:23:35.167284 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:23:35.168396 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:23:35.168402 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:23:35.169566 (+ 1164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:23:35.169580 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:23:35.170320 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:23:35.170325 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:23:35.170499 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:35.171120 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:23:35.171136 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:23:35.174375 (+ 3239us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.53.126:44263: Server oryx/127.17.53.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":58}
W20260929 20:23:35.174921 26650 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:23:35.166244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.53.67:36917 (local address 127.17.53.126:44263)
0929 20:23:35.166459 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0929 20:23:35.166464 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:23:35.166509 (+ 45us) server_negotiation.cc:407] Connection header received
0929 20:23:35.166553 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:23:35.166556 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:23:35.166613 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:23:35.166682 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:23:35.167405 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:35.168261 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:35.169712 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:35.170147 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:35.170294 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:35.174654 (+ 4360us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.53.67:36917: BlockingRecv error: recv got EOF from 127.17.53.67:36917 (error 108)
Metrics: {"server-negotiator.queue_time_us":106}
Sep 29 20:23:35 dist-test-slave-zbnc krb5kdc[25914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.53.67@KRBTEST.COM for oryx/127.17.53.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 20:23:35 dist-test-slave-zbnc krb5kdc[25914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.53.67@KRBTEST.COM for oryx/127.17.53.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 20:23:35.183708 26916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:35.175389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:44263 (local address 127.17.53.67:55823)
0929 20:23:35.175598 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0929 20:23:35.175617 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 20:23:35.175743 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:23:35.176046 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:23:35.176051 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:23:35.176349 (+ 298us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:23:35.176488 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:23:35.176495 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:23:35.177446 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:23:35.177451 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:23:35.178396 (+ 945us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:23:35.178406 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:23:35.179141 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:23:35.179147 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:23:35.179339 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:35.179878 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:23:35.179898 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:23:35.183525 (+ 3627us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.53.126:44263: Server oryx/127.17.53.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":49}
W20260929 20:23:35.184006 26914 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 20:23:35.184132 26650 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:23:35.175445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.53.67:55823 (local address 127.17.53.126:44263)
0929 20:23:35.175611 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0929 20:23:35.175617 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:23:35.175687 (+ 70us) server_negotiation.cc:407] Connection header received
0929 20:23:35.175854 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:23:35.175859 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:23:35.175923 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:23:35.176016 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:23:35.176623 (+ 607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:35.177313 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:35.178530 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:35.178994 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:35.179128 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:35.183776 (+ 4648us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.53.67:55823: BlockingRecv error: recv got EOF from 127.17.53.67:55823 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Sep 29 20:23:35 dist-test-slave-zbnc krb5kdc[25914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.53.66@KRBTEST.COM for oryx/127.17.53.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 20:23:35 dist-test-slave-zbnc krb5kdc[25914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.53.66@KRBTEST.COM for oryx/127.17.53.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 20:23:35.442503 26921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:35.433094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:44263 (local address 127.17.53.66:54187)
0929 20:23:35.433679 (+ 585us) negotiation.cc:107] Waiting for socket to connect
0929 20:23:35.433706 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0929 20:23:35.433818 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:23:35.434158 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:23:35.434161 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:23:35.434575 (+ 414us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:23:35.434720 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:23:35.434728 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:23:35.435762 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:23:35.435768 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:23:35.437055 (+ 1287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:23:35.437071 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:23:35.437855 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:23:35.437861 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:23:35.438091 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:35.438755 (+ 664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:23:35.438772 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:23:35.442270 (+ 3498us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.53.126:44263: Server oryx/127.17.53.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":478,"thread_start_us":355,"threads_started":1}
W20260929 20:23:35.442734 26650 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:23:35.433214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.53.66:54187 (local address 127.17.53.126:44263)
0929 20:23:35.433418 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0929 20:23:35.433422 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:23:35.433780 (+ 358us) server_negotiation.cc:407] Connection header received
0929 20:23:35.433949 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:23:35.433953 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:23:35.434027 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:23:35.434179 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:23:35.434839 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:35.435613 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:35.437180 (+ 1567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:35.437672 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:35.437805 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:35.442493 (+ 4688us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.53.66:54187: BlockingRecv error: recv got EOF from 127.17.53.66:54187 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 29 20:23:35 dist-test-slave-zbnc krb5kdc[25914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.53.65@KRBTEST.COM for oryx/127.17.53.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 20:23:35 dist-test-slave-zbnc krb5kdc[25914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.53.65@KRBTEST.COM for oryx/127.17.53.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 20:23:35.800220 26922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:35.791185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:44263 (local address 127.17.53.65:51645)
0929 20:23:35.791773 (+ 588us) negotiation.cc:107] Waiting for socket to connect
0929 20:23:35.791793 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 20:23:35.791909 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:23:35.792271 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:23:35.792275 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:23:35.792652 (+ 377us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:23:35.792825 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:23:35.792834 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:23:35.793800 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:23:35.793804 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:23:35.794732 (+ 928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:23:35.794743 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:23:35.795597 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:23:35.795601 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:23:35.795720 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:35.796278 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:23:35.796295 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:23:35.799953 (+ 3658us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.53.126:44263: Server oryx/127.17.53.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":356,"threads_started":1}
W20260929 20:23:35.800390 26650 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:23:35.791272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.53.65:51645 (local address 127.17.53.126:44263)
0929 20:23:35.791504 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0929 20:23:35.791509 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:23:35.791868 (+ 359us) server_negotiation.cc:407] Connection header received
0929 20:23:35.792070 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:23:35.792075 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:23:35.792140 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:23:35.792246 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:23:35.792960 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:35.793684 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:35.794876 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:35.795464 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:35.795569 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:35.800132 (+ 4563us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.53.65:51645: BlockingRecv error: recv got EOF from 127.17.53.65:51645 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
Sep 29 20:23:36 dist-test-slave-zbnc krb5kdc[25914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.53.67@KRBTEST.COM for oryx/127.17.53.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 20:23:36 dist-test-slave-zbnc krb5kdc[25914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.53.67@KRBTEST.COM for oryx/127.17.53.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 20:23:36.193043 26923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:36.184574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:44263 (local address 127.17.53.67:58491)
0929 20:23:36.185221 (+ 647us) negotiation.cc:107] Waiting for socket to connect
0929 20:23:36.185238 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 20:23:36.185346 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:23:36.185626 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:23:36.185629 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:23:36.186026 (+ 397us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:23:36.186193 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:23:36.186202 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:23:36.187281 (+ 1079us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:23:36.187285 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:23:36.188247 (+ 962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:23:36.188258 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:23:36.188871 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:23:36.188875 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:23:36.188989 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:36.189541 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:23:36.189557 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:23:36.192818 (+ 3261us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.53.126:44263: Server oryx/127.17.53.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":531,"thread_start_us":378,"threads_started":1}
W20260929 20:23:36.193243 26650 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:23:36.184735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.53.67:58491 (local address 127.17.53.126:44263)
0929 20:23:36.184934 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0929 20:23:36.184939 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:23:36.185310 (+ 371us) server_negotiation.cc:407] Connection header received
0929 20:23:36.185442 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:23:36.185446 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:23:36.185505 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:23:36.185585 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:23:36.186310 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:36.187137 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:36.188365 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:36.188726 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:36.188814 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:36.193012 (+ 4198us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.53.67:58491: BlockingRecv error: recv got EOF from 127.17.53.67:58491 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
Sep 29 20:23:36 dist-test-slave-zbnc krb5kdc[25914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.53.66@KRBTEST.COM for oryx/127.17.53.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 20:23:36 dist-test-slave-zbnc krb5kdc[25914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.53.66@KRBTEST.COM for oryx/127.17.53.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 20:23:36.453208 26924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:36.443588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:44263 (local address 127.17.53.66:46003)
0929 20:23:36.444347 (+ 759us) negotiation.cc:107] Waiting for socket to connect
0929 20:23:36.444362 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 20:23:36.444521 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:23:36.444867 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:23:36.444871 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:23:36.445331 (+ 460us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:23:36.445516 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:23:36.445525 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:23:36.446581 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:23:36.446586 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:23:36.447585 (+ 999us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:23:36.447598 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:23:36.448326 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:23:36.448332 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:23:36.448507 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:36.449187 (+ 680us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:23:36.449203 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:23:36.452849 (+ 3646us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.53.126:44263: Server oryx/127.17.53.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":630,"thread_start_us":524,"threads_started":1}
W20260929 20:23:36.453343 26650 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:23:36.443683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.53.66:46003 (local address 127.17.53.126:44263)
0929 20:23:36.443923 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0929 20:23:36.443931 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 20:23:36.444449 (+ 518us) server_negotiation.cc:407] Connection header received
0929 20:23:36.444648 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:23:36.444654 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:23:36.444729 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:23:36.444832 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:23:36.445660 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:36.446432 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:36.447729 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:36.448171 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:36.448307 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:36.453111 (+ 4804us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.53.66:46003: BlockingRecv error: recv got EOF from 127.17.53.66:46003 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Sep 29 20:23:36 dist-test-slave-zbnc krb5kdc[25914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.53.65@KRBTEST.COM for oryx/127.17.53.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 20:23:36 dist-test-slave-zbnc krb5kdc[25914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.53.65@KRBTEST.COM for oryx/127.17.53.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 20:23:36.811183 26925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:36.801262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:44263 (local address 127.17.53.65:55325)
0929 20:23:36.801925 (+ 663us) negotiation.cc:107] Waiting for socket to connect
0929 20:23:36.801947 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 20:23:36.802104 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:23:36.802455 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:23:36.802460 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:23:36.802841 (+ 381us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:23:36.803062 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:23:36.803072 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:23:36.804208 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:23:36.804212 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:23:36.805283 (+ 1071us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:23:36.805296 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:23:36.806283 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:23:36.806289 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:23:36.806585 (+ 296us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:36.807392 (+ 807us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:23:36.807409 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:23:36.810910 (+ 3501us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.53.126:44263: Server oryx/127.17.53.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":542,"thread_start_us":412,"threads_started":1}
W20260929 20:23:36.811344 26650 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:23:36.801442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.53.65:55325 (local address 127.17.53.126:44263)
0929 20:23:36.801720 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0929 20:23:36.801727 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:23:36.802028 (+ 301us) server_negotiation.cc:407] Connection header received
0929 20:23:36.802223 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:23:36.802226 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:23:36.802287 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:23:36.802372 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:23:36.803306 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:36.804030 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:36.805453 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:36.806118 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:36.806227 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:36.811134 (+ 4907us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.53.65:55325: BlockingRecv error: recv got EOF from 127.17.53.65:55325 (error 108)
Metrics: {"server-negotiator.queue_time_us":137}
Sep 29 20:23:37 dist-test-slave-zbnc krb5kdc[25914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.53.67@KRBTEST.COM for oryx/127.17.53.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 20:23:37 dist-test-slave-zbnc krb5kdc[25914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.53.67@KRBTEST.COM for oryx/127.17.53.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 20:23:37.203603 26926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:37.194180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:44263 (local address 127.17.53.67:53597)
0929 20:23:37.194735 (+ 555us) negotiation.cc:107] Waiting for socket to connect
0929 20:23:37.194752 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 20:23:37.194962 (+ 210us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:23:37.195326 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:23:37.195330 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:23:37.195796 (+ 466us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:23:37.196041 (+ 245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:23:37.196054 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:23:37.197305 (+ 1251us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:23:37.197309 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:23:37.198416 (+ 1107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:23:37.198429 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:23:37.199126 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:23:37.199132 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:23:37.199263 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:37.199866 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:23:37.199883 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:23:37.203360 (+ 3477us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.53.126:44263: Server oryx/127.17.53.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":466,"thread_start_us":321,"threads_started":1}
W20260929 20:23:37.203814 26650 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:23:37.194316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.53.67:53597 (local address 127.17.53.126:44263)
0929 20:23:37.194572 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0929 20:23:37.194579 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:23:37.194850 (+ 271us) server_negotiation.cc:407] Connection header received
0929 20:23:37.195098 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:23:37.195110 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:23:37.195189 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:23:37.195295 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:23:37.196211 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:37.197126 (+ 915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:37.198560 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:37.198984 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:37.199100 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:37.203563 (+ 4463us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.53.67:53597: BlockingRecv error: recv got EOF from 127.17.53.67:53597 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
Sep 29 20:23:37 dist-test-slave-zbnc krb5kdc[25914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.53.66@KRBTEST.COM for oryx/127.17.53.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 20:23:37 dist-test-slave-zbnc krb5kdc[25914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.53.66@KRBTEST.COM for oryx/127.17.53.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 20:23:37.463128 26927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:37.454145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:44263 (local address 127.17.53.66:45097)
0929 20:23:37.454806 (+ 661us) negotiation.cc:107] Waiting for socket to connect
0929 20:23:37.454822 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 20:23:37.455005 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:23:37.455346 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:23:37.455351 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:23:37.455732 (+ 381us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:23:37.455889 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:23:37.455898 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:23:37.456917 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:23:37.456923 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:23:37.457831 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:23:37.457843 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:23:37.458564 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:23:37.458570 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:23:37.458744 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:37.459407 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:23:37.459424 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:23:37.462767 (+ 3343us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.53.126:44263: Server oryx/127.17.53.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":573,"thread_start_us":447,"threads_started":1}
W20260929 20:23:37.463291 26650 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:23:37.454282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.53.66:45097 (local address 127.17.53.126:44263)
0929 20:23:37.454606 (+ 324us) server_negotiation.cc:206] Beginning negotiation
0929 20:23:37.454614 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 20:23:37.454924 (+ 310us) server_negotiation.cc:407] Connection header received
0929 20:23:37.455137 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:23:37.455144 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:23:37.455210 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:23:37.455311 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:23:37.456043 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:37.456775 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:37.457974 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:37.458403 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:37.458544 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:37.463036 (+ 4492us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.53.66:45097: BlockingRecv error: recv got EOF from 127.17.53.66:45097 (error 108)
Metrics: {"server-negotiator.queue_time_us":167}
Sep 29 20:23:37 dist-test-slave-zbnc krb5kdc[25914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.53.65@KRBTEST.COM for oryx/127.17.53.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 20:23:37 dist-test-slave-zbnc krb5kdc[25914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.53.65@KRBTEST.COM for oryx/127.17.53.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 20:23:37.821763 26928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:37.812341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:44263 (local address 127.17.53.65:36455)
0929 20:23:37.812950 (+ 609us) negotiation.cc:107] Waiting for socket to connect
0929 20:23:37.812978 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0929 20:23:37.813171 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:23:37.813629 (+ 458us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:23:37.813635 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:23:37.814177 (+ 542us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:23:37.814458 (+ 281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:23:37.814474 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:23:37.815680 (+ 1206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:23:37.815684 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:23:37.816676 (+ 992us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:23:37.816688 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:23:37.817330 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:23:37.817334 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:23:37.817456 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:37.818142 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:23:37.818163 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:23:37.821478 (+ 3315us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.53.126:44263: Server oryx/127.17.53.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":480,"thread_start_us":322,"threads_started":1}
W20260929 20:23:37.821925 26650 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:23:37.812542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.53.65:36455 (local address 127.17.53.126:44263)
0929 20:23:37.812777 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0929 20:23:37.812785 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 20:23:37.813126 (+ 341us) server_negotiation.cc:407] Connection header received
0929 20:23:37.813331 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:23:37.813336 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:23:37.813415 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:23:37.813525 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:23:37.814671 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:37.815534 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:37.816801 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:37.817201 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:37.817302 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:37.821698 (+ 4396us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.53.65:36455: BlockingRecv error: recv got EOF from 127.17.53.65:36455 (error 108)
Metrics: {"server-negotiator.queue_time_us":88}
Sep 29 20:23:38 dist-test-slave-zbnc krb5kdc[25914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.53.67@KRBTEST.COM for oryx/127.17.53.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 20:23:38 dist-test-slave-zbnc krb5kdc[25914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.53.67@KRBTEST.COM for oryx/127.17.53.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 20:23:38.213433 26929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:38.204722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:44263 (local address 127.17.53.67:47577)
0929 20:23:38.205174 (+ 452us) negotiation.cc:107] Waiting for socket to connect
0929 20:23:38.205190 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 20:23:38.205299 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:23:38.205611 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:23:38.205614 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:23:38.205959 (+ 345us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:23:38.206115 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:23:38.206123 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:23:38.207189 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:23:38.207193 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:23:38.208103 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:23:38.208113 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:23:38.208827 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:23:38.208832 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:23:38.209013 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:38.209666 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:23:38.209684 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:23:38.213179 (+ 3495us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.53.126:44263: Server oryx/127.17.53.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":250,"threads_started":1}
W20260929 20:23:38.213682 26650 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:23:38.204784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.53.67:47577 (local address 127.17.53.126:44263)
0929 20:23:38.205036 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0929 20:23:38.205043 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:23:38.205268 (+ 225us) server_negotiation.cc:407] Connection header received
0929 20:23:38.205398 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:23:38.205401 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:23:38.205482 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:23:38.205573 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:23:38.206265 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:38.207048 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:38.208249 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:38.208679 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:38.208780 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:38.213375 (+ 4595us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.53.67:47577: BlockingRecv error: recv got EOF from 127.17.53.67:47577 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
Sep 29 20:23:38 dist-test-slave-zbnc krb5kdc[25914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.53.66@KRBTEST.COM for oryx/127.17.53.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 20:23:38 dist-test-slave-zbnc krb5kdc[25914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.53.66@KRBTEST.COM for oryx/127.17.53.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 20:23:38.472478 26930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:38.464106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:44263 (local address 127.17.53.66:51533)
0929 20:23:38.464589 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0929 20:23:38.464605 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 20:23:38.464722 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:23:38.465009 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:23:38.465012 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:23:38.465389 (+ 377us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:23:38.465552 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:23:38.465560 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:23:38.466546 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:23:38.466549 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:23:38.467543 (+ 994us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:23:38.467554 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:23:38.468167 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:23:38.468171 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:23:38.468290 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:38.468911 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:23:38.468928 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:23:38.472225 (+ 3297us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.53.126:44263: Server oryx/127.17.53.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":394,"thread_start_us":276,"threads_started":1}
W20260929 20:23:38.472747 26650 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:23:38.464122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.53.66:51533 (local address 127.17.53.126:44263)
0929 20:23:38.464356 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0929 20:23:38.464377 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0929 20:23:38.464695 (+ 318us) server_negotiation.cc:407] Connection header received
0929 20:23:38.464820 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:23:38.464823 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:23:38.464881 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:23:38.464960 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:23:38.465690 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:38.466440 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:38.467664 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:38.468050 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:38.468141 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:38.472428 (+ 4287us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.53.66:51533: BlockingRecv error: recv got EOF from 127.17.53.66:51533 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Sep 29 20:23:38 dist-test-slave-zbnc krb5kdc[25914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.53.65@KRBTEST.COM for oryx/127.17.53.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 20:23:38 dist-test-slave-zbnc krb5kdc[25914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.53.65@KRBTEST.COM for oryx/127.17.53.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 20:23:38.831243 26931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:38.822739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:44263 (local address 127.17.53.65:37243)
0929 20:23:38.823278 (+ 539us) negotiation.cc:107] Waiting for socket to connect
0929 20:23:38.823300 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 20:23:38.823423 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:23:38.823721 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:23:38.823724 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:23:38.824158 (+ 434us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:23:38.824302 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:23:38.824311 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:23:38.825296 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:23:38.825300 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:23:38.826262 (+ 962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:23:38.826274 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:23:38.826986 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:23:38.826990 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:23:38.827107 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:38.827664 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:23:38.827682 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:23:38.830988 (+ 3306us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.53.126:44263: Server oryx/127.17.53.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":316,"threads_started":1}
W20260929 20:23:38.831430 26650 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:23:38.822790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.53.65:37243 (local address 127.17.53.126:44263)
0929 20:23:38.823052 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0929 20:23:38.823066 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 20:23:38.823401 (+ 335us) server_negotiation.cc:407] Connection header received
0929 20:23:38.823537 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:23:38.823541 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:23:38.823600 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:23:38.823676 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:23:38.824418 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:38.825181 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:38.826392 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:38.826822 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:38.826952 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:38.831183 (+ 4231us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.53.65:37243: BlockingRecv error: recv got EOF from 127.17.53.65:37243 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
Sep 29 20:23:39 dist-test-slave-zbnc krb5kdc[25914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.53.67@KRBTEST.COM for oryx/127.17.53.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 20:23:39 dist-test-slave-zbnc krb5kdc[25914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.53.67@KRBTEST.COM for oryx/127.17.53.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 20:23:39.224004 26932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:39.214697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:44263 (local address 127.17.53.67:51215)
0929 20:23:39.215316 (+ 619us) negotiation.cc:107] Waiting for socket to connect
0929 20:23:39.215332 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 20:23:39.215486 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:23:39.215837 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:23:39.215841 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:23:39.216239 (+ 398us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:23:39.216410 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:23:39.216419 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:23:39.217480 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:23:39.217488 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:23:39.218600 (+ 1112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:23:39.218611 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:23:39.219434 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:23:39.219440 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:23:39.219656 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:39.220300 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:23:39.220317 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:23:39.223763 (+ 3446us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.53.126:44263: Server oryx/127.17.53.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":532,"thread_start_us":402,"threads_started":1}
W20260929 20:23:39.224237 26650 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:23:39.214796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.53.67:51215 (local address 127.17.53.126:44263)
0929 20:23:39.215166 (+ 370us) server_negotiation.cc:206] Beginning negotiation
0929 20:23:39.215173 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:23:39.215423 (+ 250us) server_negotiation.cc:407] Connection header received
0929 20:23:39.215617 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:23:39.215622 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:23:39.215695 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:23:39.215798 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:23:39.216589 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:39.217350 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:39.218731 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:39.219293 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:39.219423 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:39.223956 (+ 4533us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.53.67:51215: BlockingRecv error: recv got EOF from 127.17.53.67:51215 (error 108)
Metrics: {"server-negotiator.queue_time_us":184}
Sep 29 20:23:39 dist-test-slave-zbnc krb5kdc[25914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.53.66@KRBTEST.COM for oryx/127.17.53.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 20:23:39 dist-test-slave-zbnc krb5kdc[25914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.53.66@KRBTEST.COM for oryx/127.17.53.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 20:23:39.491235 26933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:39.473435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:44263 (local address 127.17.53.66:57127)
0929 20:23:39.473982 (+ 547us) negotiation.cc:107] Waiting for socket to connect
0929 20:23:39.473998 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 20:23:39.474107 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:23:39.474394 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:23:39.474397 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:23:39.474774 (+ 377us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:23:39.474943 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:23:39.474952 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:23:39.475901 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:23:39.475905 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:23:39.476834 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:23:39.476845 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:23:39.477513 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:23:39.477517 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:23:39.477646 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:39.487417 (+ 9771us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:23:39.487438 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:23:39.490974 (+ 3536us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.53.126:44263: Server oryx/127.17.53.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":446,"thread_start_us":333,"threads_started":1}
W20260929 20:23:39.491626 26650 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:23:39.473581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.53.66:57127 (local address 127.17.53.126:44263)
0929 20:23:39.473834 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0929 20:23:39.473841 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:23:39.474074 (+ 233us) server_negotiation.cc:407] Connection header received
0929 20:23:39.474211 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:23:39.474215 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:23:39.474273 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:23:39.474352 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:23:39.475078 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:39.475792 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:39.476956 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:39.477377 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:39.477468 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:39.491215 (+ 13747us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.53.66:57127: BlockingRecv error: recv got EOF from 127.17.53.66:57127 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Sep 29 20:23:39 dist-test-slave-zbnc krb5kdc[25914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.53.65@KRBTEST.COM for oryx/127.17.53.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 20:23:39 dist-test-slave-zbnc krb5kdc[25914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.53.65@KRBTEST.COM for oryx/127.17.53.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 20:23:39.841348 26934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:39.832187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:44263 (local address 127.17.53.65:60915)
0929 20:23:39.832680 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0929 20:23:39.832702 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 20:23:39.832849 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:23:39.833235 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:23:39.833241 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:23:39.833662 (+ 421us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:23:39.833822 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:23:39.833831 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:23:39.834799 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:23:39.834804 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:23:39.835793 (+ 989us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:23:39.835805 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:23:39.836557 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:23:39.836564 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:23:39.836747 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:39.837444 (+ 697us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:23:39.837461 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:23:39.841055 (+ 3594us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.53.126:44263: Server oryx/127.17.53.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":373,"thread_start_us":256,"threads_started":1}
W20260929 20:23:39.841401 26650 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:23:39.832248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.53.65:60915 (local address 127.17.53.126:44263)
0929 20:23:39.832421 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0929 20:23:39.832428 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:23:39.832782 (+ 354us) server_negotiation.cc:407] Connection header received
0929 20:23:39.832982 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:23:39.832988 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:23:39.833058 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:23:39.833164 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:23:39.833964 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:39.834657 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:39.835940 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:39.836372 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:39.836510 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:39.841225 (+ 4715us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.53.65:60915: BlockingRecv error: recv got EOF from 127.17.53.65:60915 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
Sep 29 20:23:40 dist-test-slave-zbnc krb5kdc[25914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.53.67@KRBTEST.COM for oryx/127.17.53.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 20:23:40 dist-test-slave-zbnc krb5kdc[25914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.53.67@KRBTEST.COM for oryx/127.17.53.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 20:23:40.234838 26935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:40.224946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:44263 (local address 127.17.53.67:53401)
0929 20:23:40.225502 (+ 556us) negotiation.cc:107] Waiting for socket to connect
0929 20:23:40.225524 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 20:23:40.225659 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:23:40.225975 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:23:40.225978 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:23:40.226377 (+ 399us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:23:40.226528 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:23:40.226537 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:23:40.227562 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:23:40.227566 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:23:40.228529 (+ 963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:23:40.228541 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:23:40.229353 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:23:40.229357 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:23:40.229546 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:40.230286 (+ 740us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:23:40.230324 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:23:40.234581 (+ 4257us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.53.126:44263: Server oryx/127.17.53.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":428,"thread_start_us":299,"threads_started":1}
W20260929 20:23:40.235054 26650 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:23:40.224988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.53.67:53401 (local address 127.17.53.126:44263)
0929 20:23:40.225177 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0929 20:23:40.225181 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:23:40.225610 (+ 429us) server_negotiation.cc:407] Connection header received
0929 20:23:40.225759 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:23:40.225763 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:23:40.225822 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:23:40.225906 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:23:40.226653 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:40.227446 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:40.228650 (+ 1204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:40.229189 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:40.229332 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:40.234745 (+ 5413us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.53.67:53401: BlockingRecv error: recv got EOF from 127.17.53.67:53401 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Sep 29 20:23:40 dist-test-slave-zbnc krb5kdc[25914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.53.66@KRBTEST.COM for oryx/127.17.53.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 20:23:40 dist-test-slave-zbnc krb5kdc[25914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.53.66@KRBTEST.COM for oryx/127.17.53.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 20:23:40.501135 26936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:40.492222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:44263 (local address 127.17.53.66:36069)
0929 20:23:40.492900 (+ 678us) negotiation.cc:107] Waiting for socket to connect
0929 20:23:40.492917 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 20:23:40.493033 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:23:40.493441 (+ 408us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:23:40.493448 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:23:40.493929 (+ 481us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:23:40.494102 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:23:40.494111 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:23:40.495191 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:23:40.495197 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:23:40.496116 (+ 919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:23:40.496129 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:23:40.496805 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:23:40.496811 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:23:40.496981 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:40.497581 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:23:40.497599 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:23:40.500803 (+ 3204us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.53.126:44263: Server oryx/127.17.53.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":566,"thread_start_us":279,"threads_started":1}
W20260929 20:23:40.501230 26650 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:23:40.492295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.53.66:36069 (local address 127.17.53.126:44263)
0929 20:23:40.492556 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0929 20:23:40.492562 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:23:40.492998 (+ 436us) server_negotiation.cc:407] Connection header received
0929 20:23:40.493166 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:23:40.493173 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:23:40.493289 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:23:40.493400 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:23:40.494244 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:40.495052 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:40.496254 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:40.496653 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:40.496788 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:40.500986 (+ 4198us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.53.66:36069: BlockingRecv error: recv got EOF from 127.17.53.66:36069 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
Sep 29 20:23:40 dist-test-slave-zbnc krb5kdc[25914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.53.65@KRBTEST.COM for oryx/127.17.53.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 20:23:40 dist-test-slave-zbnc krb5kdc[25914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.53.65@KRBTEST.COM for oryx/127.17.53.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 20:23:40.851619 26937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:40.842599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:44263 (local address 127.17.53.65:54573)
0929 20:23:40.843254 (+ 655us) negotiation.cc:107] Waiting for socket to connect
0929 20:23:40.843275 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 20:23:40.843429 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:23:40.843889 (+ 460us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:23:40.843896 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:23:40.844478 (+ 582us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:23:40.844709 (+ 231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:23:40.844722 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:23:40.845768 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:23:40.845772 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:23:40.846701 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:23:40.846712 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:23:40.847437 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:23:40.847442 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:23:40.847553 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:40.848117 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:23:40.848140 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:23:40.851371 (+ 3231us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.53.126:44263: Server oryx/127.17.53.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":544,"thread_start_us":306,"threads_started":1}
W20260929 20:23:40.851791 26650 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:23:40.842442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.53.65:54573 (local address 127.17.53.126:44263)
0929 20:23:40.842620 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0929 20:23:40.842625 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:23:40.843380 (+ 755us) server_negotiation.cc:407] Connection header received
0929 20:23:40.843597 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:23:40.843606 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:23:40.843696 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:23:40.843847 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:23:40.844866 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:40.845649 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:40.846829 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:40.847313 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:40.847409 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:40.851577 (+ 4168us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.53.65:54573: BlockingRecv error: recv got EOF from 127.17.53.65:54573 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 29 20:23:41 dist-test-slave-zbnc krb5kdc[25914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.53.67@KRBTEST.COM for oryx/127.17.53.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 20:23:41 dist-test-slave-zbnc krb5kdc[25914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.53.67@KRBTEST.COM for oryx/127.17.53.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 20:23:41.249022 26938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:41.235898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:44263 (local address 127.17.53.67:46805)
0929 20:23:41.236369 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0929 20:23:41.236385 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 20:23:41.236570 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:23:41.236901 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:23:41.236906 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:23:41.237354 (+ 448us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:23:41.237523 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:23:41.237532 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:23:41.238597 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:23:41.238604 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:23:41.239709 (+ 1105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:23:41.239720 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:23:41.240412 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:23:41.240418 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:23:41.240604 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:41.245143 (+ 4539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:23:41.245161 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:23:41.248795 (+ 3634us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.53.126:44263: Server oryx/127.17.53.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":236,"threads_started":1}
W20260929 20:23:41.249217 26650 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:23:41.235978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.53.67:46805 (local address 127.17.53.126:44263)
0929 20:23:41.236217 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0929 20:23:41.236224 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:23:41.236500 (+ 276us) server_negotiation.cc:407] Connection header received
0929 20:23:41.236701 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:23:41.236708 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:23:41.236773 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:23:41.236872 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:23:41.237671 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:41.238450 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:41.239856 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:41.240258 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:41.240396 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:41.248992 (+ 8596us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.53.67:46805: BlockingRecv error: recv got EOF from 127.17.53.67:46805 (error 108)
Metrics: {"server-negotiator.queue_time_us":89}
Sep 29 20:23:41 dist-test-slave-zbnc krb5kdc[25914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.53.66@KRBTEST.COM for oryx/127.17.53.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 20:23:41 dist-test-slave-zbnc krb5kdc[25914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.53.66@KRBTEST.COM for oryx/127.17.53.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 20:23:41.511372 26939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:41.502028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:44263 (local address 127.17.53.66:39785)
0929 20:23:41.502572 (+ 544us) negotiation.cc:107] Waiting for socket to connect
0929 20:23:41.502588 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 20:23:41.502701 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:23:41.503097 (+ 396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:23:41.503101 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:23:41.503581 (+ 480us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:23:41.503747 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:23:41.503756 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:23:41.504741 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:23:41.504745 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:23:41.505678 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:23:41.505688 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:23:41.506368 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:23:41.506372 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:23:41.506504 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:41.507203 (+ 699us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:23:41.507220 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:23:41.511101 (+ 3881us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.53.126:44263: Server oryx/127.17.53.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":454,"thread_start_us":240,"threads_started":1}
W20260929 20:23:41.511567 26650 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:23:41.502071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.53.66:39785 (local address 127.17.53.126:44263)
0929 20:23:41.502325 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0929 20:23:41.502335 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 20:23:41.502668 (+ 333us) server_negotiation.cc:407] Connection header received
0929 20:23:41.502794 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:23:41.502798 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:23:41.502943 (+ 145us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:23:41.503029 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:23:41.503873 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:41.504611 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:41.505795 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:41.506238 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:41.506330 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:41.511286 (+ 4956us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.53.66:39785: BlockingRecv error: recv got EOF from 127.17.53.66:39785 (error 108)
Metrics: {"server-negotiator.queue_time_us":107}
Sep 29 20:23:41 dist-test-slave-zbnc krb5kdc[25914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.53.65@KRBTEST.COM for oryx/127.17.53.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 20:23:41 dist-test-slave-zbnc krb5kdc[25914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.53.65@KRBTEST.COM for oryx/127.17.53.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 20:23:41.862169 26940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:41.852822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:44263 (local address 127.17.53.65:58389)
0929 20:23:41.853340 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0929 20:23:41.853360 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 20:23:41.853484 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:23:41.853795 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:23:41.853798 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:23:41.854174 (+ 376us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:23:41.854338 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:23:41.854348 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:23:41.855376 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:23:41.855381 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:23:41.856635 (+ 1254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:23:41.856650 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:23:41.857634 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:23:41.857643 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:23:41.857820 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:41.858569 (+ 749us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:23:41.858587 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:23:41.861905 (+ 3318us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.53.126:44263: Server oryx/127.17.53.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":248,"threads_started":1}
W20260929 20:23:41.862426 26650 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:23:41.852995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.53.65:58389 (local address 127.17.53.126:44263)
0929 20:23:41.853267 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0929 20:23:41.853279 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 20:23:41.853444 (+ 165us) server_negotiation.cc:407] Connection header received
0929 20:23:41.853615 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:23:41.853618 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:23:41.853675 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:23:41.853754 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:23:41.854472 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:41.855220 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:41.856778 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:41.857389 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:41.857530 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:41.862180 (+ 4650us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.53.65:58389: BlockingRecv error: recv got EOF from 127.17.53.65:58389 (error 108)
Metrics: {"server-negotiator.queue_time_us":88}
Sep 29 20:23:42 dist-test-slave-zbnc krb5kdc[25914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.53.67@KRBTEST.COM for oryx/127.17.53.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 20:23:42 dist-test-slave-zbnc krb5kdc[25914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.53.67@KRBTEST.COM for oryx/127.17.53.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 20:23:42.259563 26941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:42.250081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:44263 (local address 127.17.53.67:45451)
0929 20:23:42.250669 (+ 588us) negotiation.cc:107] Waiting for socket to connect
0929 20:23:42.250693 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0929 20:23:42.250891 (+ 198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:23:42.251317 (+ 426us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:23:42.251324 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:23:42.251951 (+ 627us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:23:42.252121 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:23:42.252132 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:23:42.253265 (+ 1133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:23:42.253269 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:23:42.254208 (+ 939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:23:42.254219 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:23:42.255029 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:23:42.255033 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:23:42.255147 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:42.255794 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:23:42.255811 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:23:42.259338 (+ 3527us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.53.126:44263: Server oryx/127.17.53.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":447,"thread_start_us":263,"threads_started":1}
W20260929 20:23:42.259780 26650 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:23:42.250209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.53.67:45451 (local address 127.17.53.126:44263)
0929 20:23:42.250454 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0929 20:23:42.250460 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:23:42.250800 (+ 340us) server_negotiation.cc:407] Connection header received
0929 20:23:42.251033 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:23:42.251041 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:23:42.251134 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:23:42.251253 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:23:42.252290 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:42.253127 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:42.254336 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:42.254841 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:42.255006 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:42.259529 (+ 4523us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.53.67:45451: BlockingRecv error: recv got EOF from 127.17.53.67:45451 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Sep 29 20:23:42 dist-test-slave-zbnc krb5kdc[25914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.53.66@KRBTEST.COM for oryx/127.17.53.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 20:23:42 dist-test-slave-zbnc krb5kdc[25914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.53.66@KRBTEST.COM for oryx/127.17.53.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 20:23:42.521186 26942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:42.512436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:44263 (local address 127.17.53.66:53285)
0929 20:23:42.513165 (+ 729us) negotiation.cc:107] Waiting for socket to connect
0929 20:23:42.513183 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 20:23:42.513293 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:23:42.513579 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:23:42.513582 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:23:42.513948 (+ 366us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:23:42.514111 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:23:42.514118 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:23:42.515343 (+ 1225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:23:42.515346 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:23:42.516253 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:23:42.516263 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:23:42.516924 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:23:42.516928 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:23:42.517041 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:42.517628 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:23:42.517645 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:23:42.520944 (+ 3299us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.53.126:44263: Server oryx/127.17.53.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":546,"thread_start_us":343,"threads_started":1}
W20260929 20:23:42.521416 26650 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:23:42.512533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.53.66:53285 (local address 127.17.53.126:44263)
0929 20:23:42.512713 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0929 20:23:42.512719 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:23:42.513254 (+ 535us) server_negotiation.cc:407] Connection header received
0929 20:23:42.513388 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:23:42.513391 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:23:42.513465 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:23:42.513541 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:23:42.514263 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:42.515227 (+ 964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:42.516369 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:42.516778 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:42.516870 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:42.521142 (+ 4272us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.53.66:53285: BlockingRecv error: recv got EOF from 127.17.53.66:53285 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 29 20:23:42 dist-test-slave-zbnc krb5kdc[25914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.53.65@KRBTEST.COM for oryx/127.17.53.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 20:23:42 dist-test-slave-zbnc krb5kdc[25914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.53.65@KRBTEST.COM for oryx/127.17.53.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 20:23:42.871783 26943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:42.863273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:44263 (local address 127.17.53.65:58675)
0929 20:23:42.863802 (+ 529us) negotiation.cc:107] Waiting for socket to connect
0929 20:23:42.863825 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0929 20:23:42.863935 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:23:42.864213 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:23:42.864216 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:23:42.864591 (+ 375us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:23:42.864744 (+ 153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:23:42.864752 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:23:42.865684 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:23:42.865687 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:23:42.866602 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:23:42.866611 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:23:42.867290 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:23:42.867294 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:23:42.867418 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:42.867952 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:23:42.867968 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:23:42.871460 (+ 3492us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.53.126:44263: Server oryx/127.17.53.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":428,"thread_start_us":275,"threads_started":1}
W20260929 20:23:42.871886 26650 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:23:42.863286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.53.65:58675 (local address 127.17.53.126:44263)
0929 20:23:42.863477 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0929 20:23:42.863481 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:23:42.863898 (+ 417us) server_negotiation.cc:407] Connection header received
0929 20:23:42.864030 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:23:42.864033 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:23:42.864091 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:23:42.864168 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:23:42.864861 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:42.865567 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:42.866728 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:42.867164 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:42.867250 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:42.871649 (+ 4399us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.53.65:58675: BlockingRecv error: recv got EOF from 127.17.53.65:58675 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Sep 29 20:23:43 dist-test-slave-zbnc krb5kdc[25914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.53.67@KRBTEST.COM for oryx/127.17.53.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 20:23:43 dist-test-slave-zbnc krb5kdc[25914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.53.67@KRBTEST.COM for oryx/127.17.53.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 20:23:43.269330 26944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:43.260537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:44263 (local address 127.17.53.67:35873)
0929 20:23:43.261047 (+ 510us) negotiation.cc:107] Waiting for socket to connect
0929 20:23:43.261063 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 20:23:43.261244 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:23:43.261592 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:23:43.261597 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:23:43.262134 (+ 537us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:23:43.262297 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:23:43.262307 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:23:43.263369 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:23:43.263376 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:23:43.264320 (+ 944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:23:43.264332 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:23:43.265085 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:23:43.265091 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:23:43.265254 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:43.265814 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:23:43.265831 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:23:43.269093 (+ 3262us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.53.126:44263: Server oryx/127.17.53.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":267,"threads_started":1}
W20260929 20:23:43.269490 26650 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:23:43.260623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.53.67:35873 (local address 127.17.53.126:44263)
0929 20:23:43.260872 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0929 20:23:43.260878 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:23:43.261202 (+ 324us) server_negotiation.cc:407] Connection header received
0929 20:23:43.261384 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:23:43.261389 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:23:43.261458 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:23:43.261560 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:23:43.262440 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:43.263226 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:43.264462 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:43.264892 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:43.265020 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:43.269276 (+ 4256us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.53.67:35873: BlockingRecv error: recv got EOF from 127.17.53.67:35873 (error 108)
Metrics: {"server-negotiator.queue_time_us":87}
Sep 29 20:23:43 dist-test-slave-zbnc krb5kdc[25914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.53.66@KRBTEST.COM for oryx/127.17.53.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 20:23:43 dist-test-slave-zbnc krb5kdc[25914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.53.66@KRBTEST.COM for oryx/127.17.53.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 20:23:43.531441 26945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:43.522167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:44263 (local address 127.17.53.66:44079)
0929 20:23:43.522679 (+ 512us) negotiation.cc:107] Waiting for socket to connect
0929 20:23:43.522694 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 20:23:43.522849 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:23:43.523276 (+ 427us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:23:43.523280 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:23:43.523717 (+ 437us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:23:43.523852 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:23:43.523860 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:23:43.524842 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:23:43.524848 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:23:43.525979 (+ 1131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:23:43.525995 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:23:43.526989 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:23:43.526994 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:23:43.527206 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:43.527797 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:23:43.527814 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:23:43.531225 (+ 3411us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.53.126:44263: Server oryx/127.17.53.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":417,"thread_start_us":264,"threads_started":1}
W20260929 20:23:43.531741 26650 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:23:43.522225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.53.66:44079 (local address 127.17.53.126:44263)
0929 20:23:43.522456 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0929 20:23:43.522465 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 20:23:43.522779 (+ 314us) server_negotiation.cc:407] Connection header received
0929 20:23:43.523034 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:23:43.523045 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:23:43.523135 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:23:43.523244 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:23:43.523990 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:43.524704 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:43.526176 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:43.526807 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:43.526975 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:43.531445 (+ 4470us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.53.66:44079: BlockingRecv error: recv got EOF from 127.17.53.66:44079 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 29 20:23:43 dist-test-slave-zbnc krb5kdc[25914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.53.65@KRBTEST.COM for oryx/127.17.53.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 20:23:43 dist-test-slave-zbnc krb5kdc[25914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.53.65@KRBTEST.COM for oryx/127.17.53.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 20:23:43.882201 26946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:43.872886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:44263 (local address 127.17.53.65:44887)
0929 20:23:43.873446 (+ 560us) negotiation.cc:107] Waiting for socket to connect
0929 20:23:43.873467 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 20:23:43.873581 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:23:43.873868 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:23:43.873872 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:23:43.874253 (+ 381us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:23:43.874411 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:23:43.874419 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:23:43.875503 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:23:43.875507 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:23:43.876704 (+ 1197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:23:43.876719 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:23:43.877494 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:23:43.877498 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:23:43.877654 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:43.878335 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:23:43.878353 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:23:43.881823 (+ 3470us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.53.126:44263: Server oryx/127.17.53.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":331,"threads_started":1}
W20260929 20:23:43.882277 26650 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:23:43.872950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.53.65:44887 (local address 127.17.53.126:44263)
0929 20:23:43.873193 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0929 20:23:43.873197 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:23:43.873543 (+ 346us) server_negotiation.cc:407] Connection header received
0929 20:23:43.873679 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:23:43.873683 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:23:43.873744 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:23:43.873828 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:23:43.874593 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:43.875367 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:43.876855 (+ 1488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:43.877321 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:43.877474 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:43.882006 (+ 4532us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.53.65:44887: BlockingRecv error: recv got EOF from 127.17.53.65:44887 (error 108)
Metrics: {"server-negotiator.queue_time_us":119}
Sep 29 20:23:44 dist-test-slave-zbnc krb5kdc[25914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.53.67@KRBTEST.COM for oryx/127.17.53.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 20:23:44 dist-test-slave-zbnc krb5kdc[25914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.53.67@KRBTEST.COM for oryx/127.17.53.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 20:23:44.279395 26947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:44.270341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:44263 (local address 127.17.53.67:56323)
0929 20:23:44.270893 (+ 552us) negotiation.cc:107] Waiting for socket to connect
0929 20:23:44.270909 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 20:23:44.271016 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:23:44.271313 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:23:44.271316 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:23:44.271733 (+ 417us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:23:44.271875 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:23:44.271883 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:23:44.272844 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:23:44.272848 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:23:44.273837 (+ 989us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:23:44.273849 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:23:44.274483 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:23:44.274487 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:23:44.274631 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:44.275188 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:23:44.275205 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:23:44.279176 (+ 3971us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.53.126:44263: Server oryx/127.17.53.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":423,"thread_start_us":295,"threads_started":1}
W20260929 20:23:44.279588 26650 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:23:44.270504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.53.67:56323 (local address 127.17.53.126:44263)
0929 20:23:44.270708 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0929 20:23:44.270713 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:23:44.271005 (+ 292us) server_negotiation.cc:407] Connection header received
0929 20:23:44.271114 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:23:44.271118 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:23:44.271183 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:23:44.271272 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:23:44.271994 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:44.272725 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:44.273960 (+ 1235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:44.274341 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:44.274431 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:44.279345 (+ 4914us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.53.67:56323: BlockingRecv error: recv got EOF from 127.17.53.67:56323 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Sep 29 20:23:44 dist-test-slave-zbnc krb5kdc[25914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.53.66@KRBTEST.COM for oryx/127.17.53.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 20:23:44 dist-test-slave-zbnc krb5kdc[25914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.53.66@KRBTEST.COM for oryx/127.17.53.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 20:23:44.547705 26948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:44.532498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:44263 (local address 127.17.53.66:39739)
0929 20:23:44.533105 (+ 607us) negotiation.cc:107] Waiting for socket to connect
0929 20:23:44.533123 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 20:23:44.533361 (+ 238us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:23:44.533880 (+ 519us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:23:44.533888 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:23:44.534294 (+ 406us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:23:44.534537 (+ 243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:23:44.534548 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:23:44.535574 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:23:44.535578 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:23:44.536548 (+ 970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:23:44.536559 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:23:44.537209 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:23:44.537213 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:23:44.537326 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:44.543739 (+ 6413us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:23:44.543760 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:23:44.547336 (+ 3576us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.53.126:44263: Server oryx/127.17.53.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":311,"threads_started":1}
W20260929 20:23:44.547757 26650 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:23:44.532673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.53.66:39739 (local address 127.17.53.126:44263)
0929 20:23:44.532903 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0929 20:23:44.532909 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:23:44.533278 (+ 369us) server_negotiation.cc:407] Connection header received
0929 20:23:44.533599 (+ 321us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:23:44.533604 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:23:44.533699 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:23:44.533806 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:23:44.534673 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:44.535459 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:44.536664 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:44.537082 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:44.537172 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:44.547556 (+ 10384us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.53.66:39739: BlockingRecv error: recv got EOF from 127.17.53.66:39739 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
Sep 29 20:23:44 dist-test-slave-zbnc krb5kdc[25914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.53.65@KRBTEST.COM for oryx/127.17.53.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 20:23:44 dist-test-slave-zbnc krb5kdc[25914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.53.65@KRBTEST.COM for oryx/127.17.53.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 20:23:44.892639 26949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:44.883083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:44263 (local address 127.17.53.65:47559)
0929 20:23:44.883643 (+ 560us) negotiation.cc:107] Waiting for socket to connect
0929 20:23:44.883667 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0929 20:23:44.883786 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:23:44.884181 (+ 395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:23:44.884184 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:23:44.884581 (+ 397us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:23:44.884849 (+ 268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:23:44.884860 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:23:44.886184 (+ 1324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:23:44.886188 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:23:44.887166 (+ 978us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:23:44.887179 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:23:44.887804 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:23:44.887808 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:23:44.887922 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:44.888583 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:23:44.888601 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:23:44.892264 (+ 3663us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.53.126:44263: Server oryx/127.17.53.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":303,"threads_started":1}
W20260929 20:23:44.892692 26650 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:23:44.883415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.53.65:47559 (local address 127.17.53.126:44263)
0929 20:23:44.883632 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0929 20:23:44.883637 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:23:44.883763 (+ 126us) server_negotiation.cc:407] Connection header received
0929 20:23:44.883925 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:23:44.883929 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:23:44.883989 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:23:44.884079 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:23:44.885011 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:44.886053 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:44.887294 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:44.887680 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:44.887772 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:44.892457 (+ 4685us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.53.65:47559: BlockingRecv error: recv got EOF from 127.17.53.65:47559 (error 108)
Metrics: {"server-negotiator.queue_time_us":104}
Sep 29 20:23:45 dist-test-slave-zbnc krb5kdc[25914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.53.67@KRBTEST.COM for oryx/127.17.53.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 20:23:45 dist-test-slave-zbnc krb5kdc[25914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.53.67@KRBTEST.COM for oryx/127.17.53.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 20:23:45.289204 26950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:45.280269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:44263 (local address 127.17.53.67:43353)
0929 20:23:45.280950 (+ 681us) negotiation.cc:107] Waiting for socket to connect
0929 20:23:45.280966 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 20:23:45.281133 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:23:45.281431 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:23:45.281434 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:23:45.281823 (+ 389us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:23:45.281977 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:23:45.281985 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:23:45.282967 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:23:45.282971 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:23:45.283943 (+ 972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:23:45.283955 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:23:45.284643 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:23:45.284649 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:23:45.284770 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:45.285523 (+ 753us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:23:45.285541 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:23:45.288926 (+ 3385us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.53.126:44263: Server oryx/127.17.53.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":579,"thread_start_us":393,"threads_started":1}
W20260929 20:23:45.289405 26650 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:23:45.280321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.53.67:43353 (local address 127.17.53.126:44263)
0929 20:23:45.280475 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0929 20:23:45.280479 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:23:45.281090 (+ 611us) server_negotiation.cc:407] Connection header received
0929 20:23:45.281233 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:23:45.281237 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:23:45.281311 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:23:45.281392 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:23:45.282085 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:45.282785 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:45.284072 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:45.284476 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:45.284569 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:45.289155 (+ 4586us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.53.67:43353: BlockingRecv error: recv got EOF from 127.17.53.67:43353 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 29 20:23:45 dist-test-slave-zbnc krb5kdc[25914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.53.66@KRBTEST.COM for oryx/127.17.53.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 20:23:45 dist-test-slave-zbnc krb5kdc[25914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.53.66@KRBTEST.COM for oryx/127.17.53.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 20:23:45.557940 26951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:45.548744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:44263 (local address 127.17.53.66:59475)
0929 20:23:45.549227 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0929 20:23:45.549242 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 20:23:45.549396 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:23:45.549776 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:23:45.549780 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:23:45.550215 (+ 435us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:23:45.550389 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:23:45.550398 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:23:45.551595 (+ 1197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:23:45.551604 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:23:45.552517 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:23:45.552528 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:23:45.553271 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:23:45.553277 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:23:45.553452 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:45.554102 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:23:45.554120 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:23:45.557630 (+ 3510us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.53.126:44263: Server oryx/127.17.53.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":394,"thread_start_us":264,"threads_started":1}
W20260929 20:23:45.558140 26650 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:23:45.548827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.53.66:59475 (local address 127.17.53.126:44263)
0929 20:23:45.549072 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0929 20:23:45.549079 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:23:45.549327 (+ 248us) server_negotiation.cc:407] Connection header received
0929 20:23:45.549518 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:23:45.549523 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:23:45.549617 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:23:45.549718 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:23:45.550631 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:45.551464 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:45.552658 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:45.553105 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:45.553258 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:45.557824 (+ 4566us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.53.66:59475: BlockingRecv error: recv got EOF from 127.17.53.66:59475 (error 108)
Metrics: {"server-negotiator.queue_time_us":88}
Sep 29 20:23:45 dist-test-slave-zbnc krb5kdc[25914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.53.65@KRBTEST.COM for oryx/127.17.53.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 20:23:45 dist-test-slave-zbnc krb5kdc[25914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.53.65@KRBTEST.COM for oryx/127.17.53.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 20:23:45.903772 26952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:45.893874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:44263 (local address 127.17.53.65:51265)
0929 20:23:45.894537 (+ 663us) negotiation.cc:107] Waiting for socket to connect
0929 20:23:45.894561 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0929 20:23:45.894685 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:23:45.895029 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:23:45.895033 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:23:45.895529 (+ 496us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:23:45.895704 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:23:45.895713 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:23:45.896719 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:23:45.896723 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:23:45.897711 (+ 988us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:23:45.897724 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:23:45.898463 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:23:45.898469 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:23:45.898639 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:45.899504 (+ 865us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:23:45.899525 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:23:45.903399 (+ 3874us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.53.126:44263: Server oryx/127.17.53.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":556,"thread_start_us":361,"threads_started":1}
W20260929 20:23:45.903877 26650 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:23:45.893931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.53.65:51265 (local address 127.17.53.126:44263)
0929 20:23:45.894105 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0929 20:23:45.894110 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:23:45.894650 (+ 540us) server_negotiation.cc:407] Connection header received
0929 20:23:45.894789 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:23:45.894793 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:23:45.894871 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:23:45.894951 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:23:45.895844 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:45.896553 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:45.897848 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:45.898264 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:45.898362 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:45.903616 (+ 5254us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.53.65:51265: BlockingRecv error: recv got EOF from 127.17.53.65:51265 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Sep 29 20:23:46 dist-test-slave-zbnc krb5kdc[25914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.53.67@KRBTEST.COM for oryx/127.17.53.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 20:23:46 dist-test-slave-zbnc krb5kdc[25914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.53.67@KRBTEST.COM for oryx/127.17.53.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 20:23:46.298746 26953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:46.290039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:44263 (local address 127.17.53.67:36845)
0929 20:23:46.290541 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0929 20:23:46.290568 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0929 20:23:46.290679 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:23:46.291090 (+ 411us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:23:46.291093 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:23:46.291482 (+ 389us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:23:46.291652 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:23:46.291661 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:23:46.292658 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:23:46.292661 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:23:46.293586 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:23:46.293597 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:23:46.294269 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:23:46.294273 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:23:46.294388 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:46.295059 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:23:46.295076 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:23:46.298376 (+ 3300us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.53.126:44263: Server oryx/127.17.53.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":392,"thread_start_us":292,"threads_started":1}
W20260929 20:23:46.298810 26650 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:23:46.290137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.53.67:36845 (local address 127.17.53.126:44263)
0929 20:23:46.290302 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0929 20:23:46.290306 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:23:46.290646 (+ 340us) server_negotiation.cc:407] Connection header received
0929 20:23:46.290830 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:23:46.290835 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:23:46.290950 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:23:46.291047 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:23:46.291817 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:46.292544 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:46.293706 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:46.294144 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:46.294235 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:46.298568 (+ 4333us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.53.67:36845: BlockingRecv error: recv got EOF from 127.17.53.67:36845 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Sep 29 20:23:46 dist-test-slave-zbnc krb5kdc[25914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.53.66@KRBTEST.COM for oryx/127.17.53.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 20:23:46 dist-test-slave-zbnc krb5kdc[25914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.53.66@KRBTEST.COM for oryx/127.17.53.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 20:23:46.567674 26954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:46.558795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:44263 (local address 127.17.53.66:40565)
0929 20:23:46.559326 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0929 20:23:46.559342 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 20:23:46.559457 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:23:46.559756 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:23:46.559759 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:23:46.560122 (+ 363us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:23:46.560286 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:23:46.560294 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:23:46.561245 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:23:46.561249 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:23:46.562143 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:23:46.562154 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:23:46.562819 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:23:46.562823 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:23:46.562988 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:46.563638 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:23:46.563656 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:23:46.567292 (+ 3636us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.53.126:44263: Server oryx/127.17.53.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":308,"threads_started":1}
W20260929 20:23:46.567728 26650 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:23:46.558902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.53.66:40565 (local address 127.17.53.126:44263)
0929 20:23:46.559148 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0929 20:23:46.559154 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:23:46.559424 (+ 270us) server_negotiation.cc:407] Connection header received
0929 20:23:46.559557 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:23:46.559561 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:23:46.559632 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:23:46.559712 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:23:46.560404 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:46.561108 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:46.562259 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:46.562667 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:46.562757 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:46.567479 (+ 4722us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.53.66:40565: BlockingRecv error: recv got EOF from 127.17.53.66:40565 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Sep 29 20:23:46 dist-test-slave-zbnc krb5kdc[25914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.53.65@KRBTEST.COM for oryx/127.17.53.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 20:23:46 dist-test-slave-zbnc krb5kdc[25914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.53.65@KRBTEST.COM for oryx/127.17.53.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 20:23:46.913858 26955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:46.904804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:44263 (local address 127.17.53.65:52921)
0929 20:23:46.905482 (+ 678us) negotiation.cc:107] Waiting for socket to connect
0929 20:23:46.905507 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0929 20:23:46.905676 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:23:46.906030 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:23:46.906035 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:23:46.906445 (+ 410us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:23:46.906596 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:23:46.906604 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:23:46.907761 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:23:46.907767 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:23:46.908713 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:23:46.908725 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:23:46.909434 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:23:46.909441 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:23:46.909621 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:46.910165 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:23:46.910183 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:23:46.913633 (+ 3450us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.53.126:44263: Server oryx/127.17.53.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":554,"thread_start_us":446,"threads_started":1}
W20260929 20:23:46.914080 26650 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:23:46.904875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.53.65:52921 (local address 127.17.53.126:44263)
0929 20:23:46.905050 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0929 20:23:46.905055 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:23:46.905604 (+ 549us) server_negotiation.cc:407] Connection header received
0929 20:23:46.905817 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:23:46.905821 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:23:46.905886 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:23:46.905987 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:23:46.906734 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:46.907622 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:46.908863 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:46.909280 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:46.909408 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:46.913857 (+ 4449us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.53.65:52921: BlockingRecv error: recv got EOF from 127.17.53.65:52921 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 29 20:23:47 dist-test-slave-zbnc krb5kdc[25914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.53.67@KRBTEST.COM for oryx/127.17.53.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 20:23:47 dist-test-slave-zbnc krb5kdc[25914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.53.67@KRBTEST.COM for oryx/127.17.53.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 20:23:47.309203 26956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:47.299804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:44263 (local address 127.17.53.67:36941)
0929 20:23:47.300329 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0929 20:23:47.300346 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 20:23:47.300457 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:23:47.300748 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:23:47.300751 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:23:47.301152 (+ 401us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:23:47.301318 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:23:47.301327 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:23:47.302402 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:23:47.302406 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:23:47.303600 (+ 1194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:23:47.303613 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:23:47.304663 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:23:47.304670 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:23:47.304831 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:47.305509 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:23:47.305526 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:23:47.308973 (+ 3447us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.53.126:44263: Server oryx/127.17.53.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":434,"thread_start_us":281,"threads_started":1}
W20260929 20:23:47.309345 26650 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:23:47.299866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.53.67:36941 (local address 127.17.53.126:44263)
0929 20:23:47.300100 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0929 20:23:47.300109 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 20:23:47.300422 (+ 313us) server_negotiation.cc:407] Connection header received
0929 20:23:47.300561 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:23:47.300565 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:23:47.300625 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:23:47.300707 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:23:47.301447 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:47.302282 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:47.303765 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:47.304443 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:47.304572 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:47.309153 (+ 4581us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.53.67:36941: BlockingRecv error: recv got EOF from 127.17.53.67:36941 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Sep 29 20:23:47 dist-test-slave-zbnc krb5kdc[25914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.53.66@KRBTEST.COM for oryx/127.17.53.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 20:23:47 dist-test-slave-zbnc krb5kdc[25914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.53.66@KRBTEST.COM for oryx/127.17.53.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 20:23:47.578111 26957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:47.568837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:44263 (local address 127.17.53.66:56001)
0929 20:23:47.569400 (+ 563us) negotiation.cc:107] Waiting for socket to connect
0929 20:23:47.569420 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 20:23:47.569563 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:23:47.570136 (+ 573us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:23:47.570140 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:23:47.570620 (+ 480us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:23:47.570779 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:23:47.570788 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:23:47.572020 (+ 1232us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:23:47.572024 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:23:47.572931 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:23:47.572942 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:23:47.573702 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:23:47.573706 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:23:47.573824 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:47.574393 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:23:47.574410 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:23:47.577879 (+ 3469us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.53.126:44263: Server oryx/127.17.53.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":461,"thread_start_us":351,"threads_started":1}
W20260929 20:23:47.578308 26650 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:23:47.569089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.53.66:56001 (local address 127.17.53.126:44263)
0929 20:23:47.569303 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0929 20:23:47.569310 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:23:47.569519 (+ 209us) server_negotiation.cc:407] Connection header received
0929 20:23:47.569741 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:23:47.569746 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:23:47.569911 (+ 165us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:23:47.570056 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:23:47.570948 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:47.571891 (+ 943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:47.573082 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:47.573578 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:47.573674 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:47.578047 (+ 4373us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.53.66:56001: BlockingRecv error: recv got EOF from 127.17.53.66:56001 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
Sep 29 20:23:47 dist-test-slave-zbnc krb5kdc[25914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.53.65@KRBTEST.COM for oryx/127.17.53.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 20:23:47 dist-test-slave-zbnc krb5kdc[25914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.53.65@KRBTEST.COM for oryx/127.17.53.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 20:23:47.924384 26958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:47.915207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:44263 (local address 127.17.53.65:48701)
0929 20:23:47.915910 (+ 703us) negotiation.cc:107] Waiting for socket to connect
0929 20:23:47.915929 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 20:23:47.916037 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:23:47.916331 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:23:47.916335 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:23:47.916672 (+ 337us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:23:47.916851 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:23:47.916859 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:23:47.917872 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:23:47.917876 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:23:47.919034 (+ 1158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:23:47.919048 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:23:47.919807 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:23:47.919811 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:23:47.919944 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:47.920517 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:23:47.920534 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:23:47.924132 (+ 3598us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.53.126:44263: Server oryx/127.17.53.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":566,"thread_start_us":454,"threads_started":1}
W20260929 20:23:47.924583 26650 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:23:47.915285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.53.65:48701 (local address 127.17.53.126:44263)
0929 20:23:47.915521 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0929 20:23:47.915531 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 20:23:47.916015 (+ 484us) server_negotiation.cc:407] Connection header received
0929 20:23:47.916139 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:23:47.916143 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:23:47.916205 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:23:47.916288 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:23:47.916980 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:47.917737 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:47.919177 (+ 1440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:47.919680 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:47.919778 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:47.924335 (+ 4557us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.53.65:48701: BlockingRecv error: recv got EOF from 127.17.53.65:48701 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Sep 29 20:23:48 dist-test-slave-zbnc krb5kdc[25914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.53.67@KRBTEST.COM for oryx/127.17.53.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 20:23:48 dist-test-slave-zbnc krb5kdc[25914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.53.67@KRBTEST.COM for oryx/127.17.53.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 20:23:48.318305 26959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:48.310065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:44263 (local address 127.17.53.67:36555)
0929 20:23:48.310524 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0929 20:23:48.310541 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 20:23:48.310643 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:23:48.310969 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:23:48.310972 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:23:48.311387 (+ 415us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:23:48.311532 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:23:48.311541 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:23:48.312528 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:23:48.312532 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:23:48.313419 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:23:48.313430 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:23:48.314081 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:23:48.314085 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:23:48.314223 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:48.314772 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:23:48.314788 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:23:48.317997 (+ 3209us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.53.126:44263: Server oryx/127.17.53.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":351,"thread_start_us":233,"threads_started":1}
W20260929 20:23:48.318554 26650 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:23:48.310288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.53.67:36555 (local address 127.17.53.126:44263)
0929 20:23:48.310585 (+ 297us) server_negotiation.cc:206] Beginning negotiation
0929 20:23:48.310590 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:23:48.310606 (+ 16us) server_negotiation.cc:407] Connection header received
0929 20:23:48.310745 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:23:48.310749 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:23:48.310806 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:23:48.310941 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:23:48.311660 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:48.312403 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:48.313569 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:48.313967 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:48.314056 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:48.318268 (+ 4212us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.53.67:36555: BlockingRecv error: recv got EOF from 127.17.53.67:36555 (error 108)
Metrics: {"server-negotiator.queue_time_us":151}
Sep 29 20:23:48 dist-test-slave-zbnc krb5kdc[25914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.53.66@KRBTEST.COM for oryx/127.17.53.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 20:23:48 dist-test-slave-zbnc krb5kdc[25914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.53.66@KRBTEST.COM for oryx/127.17.53.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 20:23:48.589721 26960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:48.579429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:44263 (local address 127.17.53.66:37655)
0929 20:23:48.579922 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0929 20:23:48.579938 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 20:23:48.580090 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:23:48.580437 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:23:48.580441 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:23:48.580880 (+ 439us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:23:48.581106 (+ 226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:23:48.581119 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:23:48.582166 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:23:48.582174 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:23:48.583551 (+ 1377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:23:48.583572 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:23:48.584536 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:23:48.584543 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:23:48.584773 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:48.585478 (+ 705us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:23:48.585499 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:23:48.589383 (+ 3884us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.53.126:44263: Server oryx/127.17.53.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":274,"threads_started":1}
W20260929 20:23:48.589882 26650 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:23:48.579509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.53.66:37655 (local address 127.17.53.126:44263)
0929 20:23:48.579705 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0929 20:23:48.579713 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 20:23:48.580023 (+ 310us) server_negotiation.cc:407] Connection header received
0929 20:23:48.580212 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:23:48.580221 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:23:48.580292 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:23:48.580407 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:23:48.581253 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:48.582021 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:48.583761 (+ 1740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:48.584300 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:48.584526 (+ 226us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:48.589634 (+ 5108us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.53.66:37655: BlockingRecv error: recv got EOF from 127.17.53.66:37655 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 29 20:23:48 dist-test-slave-zbnc krb5kdc[25914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.53.65@KRBTEST.COM for oryx/127.17.53.126@KRBTEST.COM, Server not found in Kerberos database
Sep 29 20:23:48 dist-test-slave-zbnc krb5kdc[25914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.17.53.65@KRBTEST.COM for oryx/127.17.53.126@KRBTEST.COM, Server not found in Kerberos database
I20260929 20:23:48.934540 26961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:48.925383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:44263 (local address 127.17.53.65:60309)
0929 20:23:48.925890 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0929 20:23:48.925910 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 20:23:48.926027 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:23:48.926319 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:23:48.926322 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:23:48.926703 (+ 381us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:23:48.926896 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:23:48.926905 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:23:48.927897 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:23:48.927901 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:23:48.928822 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:23:48.928834 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:23:48.929745 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:23:48.929751 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:23:48.929926 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:48.930756 (+ 830us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:23:48.930778 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:23:48.934197 (+ 3419us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.17.53.126:44263: Server oryx/127.17.53.126@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":417,"thread_start_us":311,"threads_started":1}
W20260929 20:23:48.934691 26650 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:23:48.925484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.53.65:60309 (local address 127.17.53.126:44263)
0929 20:23:48.925753 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0929 20:23:48.925760 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:23:48.926001 (+ 241us) server_negotiation.cc:407] Connection header received
0929 20:23:48.926132 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:23:48.926136 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:23:48.926197 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:23:48.926280 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:23:48.927036 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:48.927772 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:48.928961 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:48.929579 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:48.929697 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:48.934438 (+ 4741us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.17.53.65:60309: BlockingRecv error: recv got EOF from 127.17.53.65:60309 (error 108)
Metrics: {"server-negotiator.queue_time_us":100}
I20260929 20:23:49.052677 17621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu pid 26501
I20260929 20:23:49.067984 26963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:49.054174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44190 (local address 127.17.53.65:37931)
0929 20:23:49.055697 (+ 1523us) server_negotiation.cc:206] Beginning negotiation
0929 20:23:49.055707 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 20:23:49.055747 (+ 40us) server_negotiation.cc:407] Connection header received
0929 20:23:49.055827 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:23:49.055835 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:23:49.056170 (+ 335us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:23:49.056452 (+ 282us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:23:49.058342 (+ 1890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:49.059648 (+ 1306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:49.060911 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:49.061508 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:49.061661 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:49.064399 (+ 2738us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:23:49.064407 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:23:49.064430 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:23:49.064468 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:23:49.067252 (+ 2784us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.17.53.65@KRBTEST.COM not found in keytab (ticket kvno 2)
0929 20:23:49.067649 (+ 397us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:44190: Request ticket server kudu/127.17.53.65@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":1278,"thread_start_us":1120,"threads_started":1}
W20260929 20:23:49.068099 26963 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:44190: Request ticket server kudu/127.17.53.65@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260929 20:23:49.068063 17621 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu pid 26501: Not authorized: Client connection negotiation failed: client connection to 127.17.53.65:37931: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.17.53.65@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260929 20:23:49.068279 17621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu with pid 26501
I20260929 20:23:49.085692 17621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu pid 26652
I20260929 20:23:49.100164 26964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:49.087038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40850 (local address 127.17.53.66:40543)
0929 20:23:49.087736 (+ 698us) server_negotiation.cc:206] Beginning negotiation
0929 20:23:49.087743 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:23:49.087775 (+ 32us) server_negotiation.cc:407] Connection header received
0929 20:23:49.087837 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:23:49.087845 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:23:49.088099 (+ 254us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:23:49.088293 (+ 194us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:23:49.090232 (+ 1939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:49.091375 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:49.092821 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:49.093431 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:49.093560 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:49.096414 (+ 2854us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:23:49.096422 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:23:49.096438 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:23:49.096484 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:23:49.099583 (+ 3099us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.17.53.66@KRBTEST.COM not found in keytab (ticket kvno 2)
0929 20:23:49.099924 (+ 341us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:40850: Request ticket server kudu/127.17.53.66@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":508,"thread_start_us":365,"threads_started":1}
W20260929 20:23:49.100306 26964 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:40850: Request ticket server kudu/127.17.53.66@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260929 20:23:49.100425 17621 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu pid 26652: Not authorized: Client connection negotiation failed: client connection to 127.17.53.66:40543: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.17.53.66@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260929 20:23:49.100579 17621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu with pid 26652
I20260929 20:23:49.117779 17621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu pid 26785
I20260929 20:23:49.131218 26965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:49.119149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39510 (local address 127.17.53.67:39981)
0929 20:23:49.119721 (+ 572us) server_negotiation.cc:206] Beginning negotiation
0929 20:23:49.119728 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:23:49.119754 (+ 26us) server_negotiation.cc:407] Connection header received
0929 20:23:49.119819 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:23:49.119826 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:23:49.120064 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:23:49.120239 (+ 175us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:23:49.122262 (+ 2023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:49.123375 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:49.124630 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:49.125220 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:49.125323 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:49.128045 (+ 2722us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:23:49.128053 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:23:49.128068 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:23:49.128115 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:23:49.130568 (+ 2453us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.17.53.67@KRBTEST.COM not found in keytab (ticket kvno 2)
0929 20:23:49.130927 (+ 359us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:39510: Request ticket server kudu/127.17.53.67@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":392,"thread_start_us":258,"threads_started":1}
W20260929 20:23:49.131335 26965 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:39510: Request ticket server kudu/127.17.53.67@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260929 20:23:49.131413 17621 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu pid 26785: Not authorized: Client connection negotiation failed: client connection to 127.17.53.67:39981: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.17.53.67@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260929 20:23:49.131562 17621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu with pid 26785
I20260929 20:23:49.150321 17621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu pid 26432
I20260929 20:23:49.151832 26494 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:23:49.269901 17621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu with pid 26432
2026-09-29T20:23:49Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (21634 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-29T20:23:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T20:23:49Z Disabled control of system clock
I20260929 20:23:49.346540 17621 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
/tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790713339161492-17621-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790713339161492-17621-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790713339161492-17621-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790713339161492-17621-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.53.126:44889
--webserver_interface=127.17.53.126
--webserver_port=0
--builtin_ntp_servers=127.17.53.84:35733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.53.126:44889
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260929 20:23:49.679159 26970 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:23:49.679451 26970 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:23:49.679615 26970 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:23:49.686916 26970 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:23:49.687021 26970 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:23:49.687106 26970 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:23:49.687215 26970 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:23:49.687294 26970 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:23:49.696594 26970 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.53.84:35733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790713339161492-17621-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790713339161492-17621-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.53.126:44889
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.53.126:44889
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790713339161492-17621-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.53.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790713339161492-17621-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:23:49.697978 26970 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:23:49.699828 26970 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:23:49.711954 26978 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:23:49.711977 26976 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:23:49.712188 26975 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:23:49.712898 26970 server_base.cc:1034] running on GCE node
I20260929 20:23:49.713691 26970 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:23:49.715130 26970 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:23:49.716325 26970 hybrid_clock.cc:648] HybridClock initialized: now 1790713429716274 us; error 39 us; skew 500 ppm
I20260929 20:23:49.716703 26970 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 20:23:49.723820 26970 webserver.cc:460] Webserver started at http://127.17.53.126:43941/ using document root <none> and password file <none>
I20260929 20:23:49.724601 26970 fs_manager.cc:359] Metadata directory not provided
I20260929 20:23:49.724720 26970 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:23:49.725071 26970 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:23:49.728063 26970 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790713339161492-17621-0/minicluster-data/master-0/data/instance:
uuid: "417b2148852b440e8bca242e8b91bd6c"
format_stamp: "Formatted at 2026-09-29 20:23:49 on dist-test-slave-zbnc"
server_key: "f9f8661327aedeabb2afd339d0587e74"
server_key_iv: "186a970ac584891a749910efe1c2fa75"
server_key_version: "encryptionkey@0"
I20260929 20:23:49.728857 26970 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790713339161492-17621-0/minicluster-data/master-0/wal/instance:
uuid: "417b2148852b440e8bca242e8b91bd6c"
format_stamp: "Formatted at 2026-09-29 20:23:49 on dist-test-slave-zbnc"
server_key: "f9f8661327aedeabb2afd339d0587e74"
server_key_iv: "186a970ac584891a749910efe1c2fa75"
server_key_version: "encryptionkey@0"
I20260929 20:23:49.734126 26970 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 20:23:49.737531 26984 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:23:49.738919 26970 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 20:23:49.739095 26970 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790713339161492-17621-0/minicluster-data/master-0/data,/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790713339161492-17621-0/minicluster-data/master-0/wal
uuid: "417b2148852b440e8bca242e8b91bd6c"
format_stamp: "Formatted at 2026-09-29 20:23:49 on dist-test-slave-zbnc"
server_key: "f9f8661327aedeabb2afd339d0587e74"
server_key_iv: "186a970ac584891a749910efe1c2fa75"
server_key_version: "encryptionkey@0"
I20260929 20:23:49.739315 26970 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790713339161492-17621-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790713339161492-17621-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790713339161492-17621-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:23:49.751204 26970 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:23:49.752020 26970 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:23:49.752321 26970 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:23:49.770507 26970 rpc_server.cc:307] RPC server started. Bound to: 127.17.53.126:44889
I20260929 20:23:49.770510 27035 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.53.126:44889 every 8 connection(s)
I20260929 20:23:49.771888 26970 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790713339161492-17621-0/minicluster-data/master-0/data/info.pb
I20260929 20:23:49.775808 27036 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:23:49.778023 17621 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu as pid 26970
I20260929 20:23:49.778214 17621 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790713339161492-17621-0/minicluster-data/master-0/wal/instance
I20260929 20:23:49.778918 17621 external_mini_cluster.cc:1428] Setting key d3d24c390d84f4819885f913fa72545e
I20260929 20:23:49.789239 27036 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 417b2148852b440e8bca242e8b91bd6c: Bootstrap starting.
I20260929 20:23:49.793213 27036 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 417b2148852b440e8bca242e8b91bd6c: Neither blocks nor log segments found. Creating new log.
I20260929 20:23:49.794590 27036 log.cc:824] T 00000000000000000000000000000000 P 417b2148852b440e8bca242e8b91bd6c: Log is configured to *not* fsync() on all Append() calls
I20260929 20:23:49.795346 27039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:49.781121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56540 (local address 127.17.53.126:44889)
0929 20:23:49.782267 (+ 1146us) server_negotiation.cc:206] Beginning negotiation
0929 20:23:49.782284 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0929 20:23:49.782336 (+ 52us) server_negotiation.cc:407] Connection header received
0929 20:23:49.784615 (+ 2279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:23:49.784657 (+ 42us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:23:49.785267 (+ 610us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:23:49.785728 (+ 461us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:23:49.786282 (+ 554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:49.788310 (+ 2028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:49.789886 (+ 1576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:49.790975 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:49.791164 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:49.792313 (+ 1149us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:23:49.792323 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:23:49.792339 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:23:49.793039 (+ 700us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:23:49.793195 (+ 156us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:23:49.793229 (+ 34us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:23:49.793337 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0929 20:23:49.793588 (+ 251us) server_negotiation.cc:299] Negotiation successful
0929 20:23:49.793950 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":637,"thread_start_us":402,"threads_started":1}
I20260929 20:23:49.797662 27036 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 417b2148852b440e8bca242e8b91bd6c: No bootstrap required, opened a new log
I20260929 20:23:49.804989 27036 raft_consensus.cc:348] T 00000000000000000000000000000000 P 417b2148852b440e8bca242e8b91bd6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "417b2148852b440e8bca242e8b91bd6c" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 44889 } }
I20260929 20:23:49.805373 27036 raft_consensus.cc:374] T 00000000000000000000000000000000 P 417b2148852b440e8bca242e8b91bd6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:23:49.805505 27036 raft_consensus.cc:729] T 00000000000000000000000000000000 P 417b2148852b440e8bca242e8b91bd6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 417b2148852b440e8bca242e8b91bd6c, State: Initialized, Role: FOLLOWER
I20260929 20:23:49.806080 27036 consensus_queue.cc:260] T 00000000000000000000000000000000 P 417b2148852b440e8bca242e8b91bd6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "417b2148852b440e8bca242e8b91bd6c" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 44889 } }
I20260929 20:23:49.806301 27036 raft_consensus.cc:388] T 00000000000000000000000000000000 P 417b2148852b440e8bca242e8b91bd6c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 20:23:49.806440 27036 raft_consensus.cc:482] T 00000000000000000000000000000000 P 417b2148852b440e8bca242e8b91bd6c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 20:23:49.806603 27036 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 417b2148852b440e8bca242e8b91bd6c [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:23:49.808849 27036 raft_consensus.cc:504] T 00000000000000000000000000000000 P 417b2148852b440e8bca242e8b91bd6c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "417b2148852b440e8bca242e8b91bd6c" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 44889 } }
I20260929 20:23:49.809381 27036 leader_election.cc:302] T 00000000000000000000000000000000 P 417b2148852b440e8bca242e8b91bd6c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 417b2148852b440e8bca242e8b91bd6c; no voters:
I20260929 20:23:49.810076 27036 leader_election.cc:288] T 00000000000000000000000000000000 P 417b2148852b440e8bca242e8b91bd6c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 20:23:49.810263 27041 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 417b2148852b440e8bca242e8b91bd6c [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:23:49.811024 27041 raft_consensus.cc:686] T 00000000000000000000000000000000 P 417b2148852b440e8bca242e8b91bd6c [term 1 LEADER]: Becoming Leader. State: Replica: 417b2148852b440e8bca242e8b91bd6c, State: Running, Role: LEADER
I20260929 20:23:49.811573 27041 consensus_queue.cc:237] T 00000000000000000000000000000000 P 417b2148852b440e8bca242e8b91bd6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "417b2148852b440e8bca242e8b91bd6c" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 44889 } }
I20260929 20:23:49.812240 27036 sys_catalog.cc:564] T 00000000000000000000000000000000 P 417b2148852b440e8bca242e8b91bd6c [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:23:49.814677 27043 sys_catalog.cc:455] T 00000000000000000000000000000000 P 417b2148852b440e8bca242e8b91bd6c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 417b2148852b440e8bca242e8b91bd6c. Latest consensus state: current_term: 1 leader_uuid: "417b2148852b440e8bca242e8b91bd6c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "417b2148852b440e8bca242e8b91bd6c" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 44889 } } }
I20260929 20:23:49.814567 27042 sys_catalog.cc:455] T 00000000000000000000000000000000 P 417b2148852b440e8bca242e8b91bd6c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "417b2148852b440e8bca242e8b91bd6c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "417b2148852b440e8bca242e8b91bd6c" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 44889 } } }
I20260929 20:23:49.815066 27043 sys_catalog.cc:458] T 00000000000000000000000000000000 P 417b2148852b440e8bca242e8b91bd6c [sys.catalog]: This master's current role is: LEADER
I20260929 20:23:49.815105 27042 sys_catalog.cc:458] T 00000000000000000000000000000000 P 417b2148852b440e8bca242e8b91bd6c [sys.catalog]: This master's current role is: LEADER
I20260929 20:23:49.817137 27050 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:23:49.823086 27050 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:23:49.832376 27050 catalog_manager.cc:1347] Generated new cluster ID: ec93972cd2dc43ea8be78276f1a042b8
I20260929 20:23:49.832531 27050 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:23:49.858107 27050 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 20:23:49.859643 27050 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:23:49.872056 27050 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 417b2148852b440e8bca242e8b91bd6c: Generated new TSK 0
I20260929 20:23:49.873107 27050 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 20:23:49.882121 17621 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
/tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790713339161492-17621-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790713339161492-17621-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790713339161492-17621-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790713339161492-17621-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.53.65:0
--local_ip_for_outbound_sockets=127.17.53.65
--webserver_interface=127.17.53.65
--webserver_port=0
--tserver_master_addrs=127.17.53.126:44889
--builtin_ntp_servers=127.17.53.84:35733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260929 20:23:50.222327 27060 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:23:50.222704 27060 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:23:50.222820 27060 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:23:50.230154 27060 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:23:50.230262 27060 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:23:50.230446 27060 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.53.65
I20260929 20:23:50.239826 27060 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.53.84:35733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790713339161492-17621-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790713339161492-17621-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.53.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790713339161492-17621-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.17.53.65
--webserver_port=0
--tserver_master_addrs=127.17.53.126:44889
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.53.65
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790713339161492-17621-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:23:50.241139 27060 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:23:50.243075 27060 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:23:50.256008 27065 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:23:50.256093 27068 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:23:50.256040 27066 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:23:50.256762 27060 server_base.cc:1034] running on GCE node
I20260929 20:23:50.257401 27060 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:23:50.258335 27060 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:23:50.259598 27060 hybrid_clock.cc:648] HybridClock initialized: now 1790713430259545 us; error 43 us; skew 500 ppm
I20260929 20:23:50.259938 27060 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 20:23:50.262290 27060 webserver.cc:460] Webserver started at http://127.17.53.65:39633/ using document root <none> and password file <none>
I20260929 20:23:50.263068 27060 fs_manager.cc:359] Metadata directory not provided
I20260929 20:23:50.263206 27060 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:23:50.263540 27060 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:23:50.266381 27060 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790713339161492-17621-0/minicluster-data/ts-0/data/instance:
uuid: "f10e9e32be0343da80beb37411d8264c"
format_stamp: "Formatted at 2026-09-29 20:23:50 on dist-test-slave-zbnc"
server_key: "15946516357c00d2bd082c3fdbf8aa0b"
server_key_iv: "5c808dede39f6fa10572277506e6afa0"
server_key_version: "encryptionkey@0"
I20260929 20:23:50.267253 27060 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790713339161492-17621-0/minicluster-data/ts-0/wal/instance:
uuid: "f10e9e32be0343da80beb37411d8264c"
format_stamp: "Formatted at 2026-09-29 20:23:50 on dist-test-slave-zbnc"
server_key: "15946516357c00d2bd082c3fdbf8aa0b"
server_key_iv: "5c808dede39f6fa10572277506e6afa0"
server_key_version: "encryptionkey@0"
I20260929 20:23:50.272342 27060 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260929 20:23:50.275792 27074 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:23:50.277122 27060 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 20:23:50.277295 27060 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790713339161492-17621-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790713339161492-17621-0/minicluster-data/ts-0/wal
uuid: "f10e9e32be0343da80beb37411d8264c"
format_stamp: "Formatted at 2026-09-29 20:23:50 on dist-test-slave-zbnc"
server_key: "15946516357c00d2bd082c3fdbf8aa0b"
server_key_iv: "5c808dede39f6fa10572277506e6afa0"
server_key_version: "encryptionkey@0"
I20260929 20:23:50.277539 27060 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790713339161492-17621-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790713339161492-17621-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790713339161492-17621-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:23:50.315457 27060 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:23:50.316545 27060 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:23:50.316845 27060 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:23:50.317824 27060 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:23:50.319381 27060 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:23:50.319645 27060 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:23:50.319773 27060 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:23:50.319862 27060 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:23:50.353816 27060 rpc_server.cc:307] RPC server started. Bound to: 127.17.53.65:38025
I20260929 20:23:50.353870 27186 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.53.65:38025 every 8 connection(s)
I20260929 20:23:50.355283 27060 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790713339161492-17621-0/minicluster-data/ts-0/data/info.pb
I20260929 20:23:50.365355 17621 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu as pid 27060
I20260929 20:23:50.365568 17621 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790713339161492-17621-0/minicluster-data/ts-0/wal/instance
I20260929 20:23:50.366303 17621 external_mini_cluster.cc:1428] Setting key 3fbe4f3c1f562af897220615f1d28021
I20260929 20:23:50.369124 27189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:50.357852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.53.65:55019 (local address 127.17.53.126:44889)
0929 20:23:50.358293 (+ 441us) server_negotiation.cc:206] Beginning negotiation
0929 20:23:50.358297 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:23:50.359625 (+ 1328us) server_negotiation.cc:407] Connection header received
0929 20:23:50.361474 (+ 1849us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:23:50.361481 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:23:50.361579 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:23:50.361699 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:23:50.362759 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:50.363582 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:50.365975 (+ 2393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:50.366518 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:50.366637 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:50.368290 (+ 1653us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:23:50.368297 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:23:50.368302 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:23:50.368334 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:23:50.368402 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:23:50.368423 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:23:50.368521 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0929 20:23:50.368801 (+ 280us) server_negotiation.cc:299] Negotiation successful
0929 20:23:50.368875 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":337,"thread_start_us":239,"threads_started":1}
I20260929 20:23:50.369546 17621 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
/tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790713339161492-17621-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790713339161492-17621-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790713339161492-17621-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790713339161492-17621-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.53.66:0
--local_ip_for_outbound_sockets=127.17.53.66
--webserver_interface=127.17.53.66
--webserver_port=0
--tserver_master_addrs=127.17.53.126:44889
--builtin_ntp_servers=127.17.53.84:35733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260929 20:23:50.370565 27190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:50.358491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:44889 (local address 127.17.53.65:55019)
0929 20:23:50.359462 (+ 971us) negotiation.cc:107] Waiting for socket to connect
0929 20:23:50.359506 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 20:23:50.361206 (+ 1700us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:23:50.361933 (+ 727us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:23:50.361941 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:23:50.362036 (+ 95us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:23:50.362597 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:23:50.362613 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:23:50.363744 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:23:50.363749 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:23:50.365802 (+ 2053us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:23:50.365816 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:23:50.366751 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:23:50.366759 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:23:50.367081 (+ 322us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:50.367923 (+ 842us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 20:23:50.367925 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:23:50.367963 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 20:23:50.368067 (+ 104us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:23:50.368548 (+ 481us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:23:50.368561 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:23:50.368569 (+ 8us) client_negotiation.cc:769] Sending connection context
0929 20:23:50.368758 (+ 189us) client_negotiation.cc:240] Negotiation successful
0929 20:23:50.369161 (+ 403us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":524,"thread_start_us":324,"threads_started":1}
I20260929 20:23:50.372270 27187 heartbeater.cc:347] Connected to a master server at 127.17.53.126:44889
I20260929 20:23:50.372581 27187 heartbeater.cc:464] Registering TS with master...
I20260929 20:23:50.373363 27187 heartbeater.cc:511] Master 127.17.53.126:44889 requested a full tablet report, sending...
I20260929 20:23:50.375687 27001 ts_manager.cc:194] Registered new tserver with Master: f10e9e32be0343da80beb37411d8264c (127.17.53.65:38025)
I20260929 20:23:50.377359 27001 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.17.53.65:55019
W20260929 20:23:50.701004 27191 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:23:50.701339 27191 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:23:50.701488 27191 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:23:50.708662 27191 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:23:50.708807 27191 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:23:50.708992 27191 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.53.66
I20260929 20:23:50.718555 27191 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.53.84:35733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790713339161492-17621-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790713339161492-17621-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.53.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790713339161492-17621-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.17.53.66
--webserver_port=0
--tserver_master_addrs=127.17.53.126:44889
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.53.66
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790713339161492-17621-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:23:50.719926 27191 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:23:50.722100 27191 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:23:50.735986 27196 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:23:50.735950 27197 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:23:50.736081 27199 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:23:50.736806 27191 server_base.cc:1034] running on GCE node
I20260929 20:23:50.737402 27191 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:23:50.738364 27191 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:23:50.739621 27191 hybrid_clock.cc:648] HybridClock initialized: now 1790713430739562 us; error 50 us; skew 500 ppm
I20260929 20:23:50.739962 27191 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 20:23:50.742333 27191 webserver.cc:460] Webserver started at http://127.17.53.66:34093/ using document root <none> and password file <none>
I20260929 20:23:50.743183 27191 fs_manager.cc:359] Metadata directory not provided
I20260929 20:23:50.743307 27191 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:23:50.743649 27191 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:23:50.746541 27191 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790713339161492-17621-0/minicluster-data/ts-1/data/instance:
uuid: "966b1ef3a2cd4513a603fc8922e6b1ef"
format_stamp: "Formatted at 2026-09-29 20:23:50 on dist-test-slave-zbnc"
server_key: "5c5c87a2d235c51f96b17d04a6f67c69"
server_key_iv: "82132a11328897203214df9d4d7fbe4a"
server_key_version: "encryptionkey@0"
I20260929 20:23:50.747432 27191 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790713339161492-17621-0/minicluster-data/ts-1/wal/instance:
uuid: "966b1ef3a2cd4513a603fc8922e6b1ef"
format_stamp: "Formatted at 2026-09-29 20:23:50 on dist-test-slave-zbnc"
server_key: "5c5c87a2d235c51f96b17d04a6f67c69"
server_key_iv: "82132a11328897203214df9d4d7fbe4a"
server_key_version: "encryptionkey@0"
I20260929 20:23:50.752841 27191 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 20:23:50.756368 27205 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:23:50.757737 27191 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 20:23:50.757927 27191 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790713339161492-17621-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790713339161492-17621-0/minicluster-data/ts-1/wal
uuid: "966b1ef3a2cd4513a603fc8922e6b1ef"
format_stamp: "Formatted at 2026-09-29 20:23:50 on dist-test-slave-zbnc"
server_key: "5c5c87a2d235c51f96b17d04a6f67c69"
server_key_iv: "82132a11328897203214df9d4d7fbe4a"
server_key_version: "encryptionkey@0"
I20260929 20:23:50.758152 27191 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790713339161492-17621-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790713339161492-17621-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790713339161492-17621-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:23:50.805650 27191 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:23:50.806617 27191 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:23:50.806957 27191 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:23:50.808025 27191 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:23:50.809612 27191 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:23:50.809698 27191 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:23:50.809849 27191 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:23:50.809922 27191 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:23:50.844396 27191 rpc_server.cc:307] RPC server started. Bound to: 127.17.53.66:45541
I20260929 20:23:50.844449 27317 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.53.66:45541 every 8 connection(s)
I20260929 20:23:50.845752 27191 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790713339161492-17621-0/minicluster-data/ts-1/data/info.pb
I20260929 20:23:50.854321 17621 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu as pid 27191
I20260929 20:23:50.854496 17621 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790713339161492-17621-0/minicluster-data/ts-1/wal/instance
I20260929 20:23:50.855084 17621 external_mini_cluster.cc:1428] Setting key 7676ad88f81fef35bc9b572e8cdc5643
I20260929 20:23:50.857898 17621 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
/tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790713339161492-17621-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790713339161492-17621-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790713339161492-17621-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790713339161492-17621-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.53.67:0
--local_ip_for_outbound_sockets=127.17.53.67
--webserver_interface=127.17.53.67
--webserver_port=0
--tserver_master_addrs=127.17.53.126:44889
--builtin_ntp_servers=127.17.53.84:35733
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260929 20:23:50.858820 27189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:50.848374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.53.66:40719 (local address 127.17.53.126:44889)
0929 20:23:50.848563 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0929 20:23:50.848567 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:23:50.850061 (+ 1494us) server_negotiation.cc:407] Connection header received
0929 20:23:50.851877 (+ 1816us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:23:50.851881 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:23:50.851947 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:23:50.852057 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:23:50.853098 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:50.853811 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:50.856026 (+ 2215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:50.856457 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:50.856564 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:50.858027 (+ 1463us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:23:50.858033 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:23:50.858039 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:23:50.858061 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:23:50.858104 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:23:50.858123 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:23:50.858216 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0929 20:23:50.858527 (+ 311us) server_negotiation.cc:299] Negotiation successful
0929 20:23:50.858604 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260929 20:23:50.860466 27320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:50.849007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:44889 (local address 127.17.53.66:40719)
0929 20:23:50.849876 (+ 869us) negotiation.cc:107] Waiting for socket to connect
0929 20:23:50.849918 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 20:23:50.851611 (+ 1693us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:23:50.852269 (+ 658us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:23:50.852278 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:23:50.852368 (+ 90us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:23:50.852941 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:23:50.852957 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:23:50.853961 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:23:50.853966 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:23:50.855886 (+ 1920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:23:50.855898 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:23:50.856690 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:23:50.856697 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:23:50.856999 (+ 302us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:50.857713 (+ 714us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 20:23:50.857716 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:23:50.857757 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 20:23:50.857833 (+ 76us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:23:50.858270 (+ 437us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:23:50.858284 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:23:50.858295 (+ 11us) client_negotiation.cc:769] Sending connection context
0929 20:23:50.858513 (+ 218us) client_negotiation.cc:240] Negotiation successful
0929 20:23:50.858996 (+ 483us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":270,"threads_started":1}
I20260929 20:23:50.861922 27318 heartbeater.cc:347] Connected to a master server at 127.17.53.126:44889
I20260929 20:23:50.862229 27318 heartbeater.cc:464] Registering TS with master...
I20260929 20:23:50.863129 27318 heartbeater.cc:511] Master 127.17.53.126:44889 requested a full tablet report, sending...
I20260929 20:23:50.864529 27001 ts_manager.cc:194] Registered new tserver with Master: 966b1ef3a2cd4513a603fc8922e6b1ef (127.17.53.66:45541)
I20260929 20:23:50.865506 27001 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.17.53.66:40719
W20260929 20:23:51.196125 27321 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:23:51.196416 27321 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:23:51.196504 27321 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:23:51.203742 27321 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:23:51.203850 27321 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:23:51.203953 27321 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.53.67
I20260929 20:23:51.213791 27321 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.53.84:35733
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790713339161492-17621-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790713339161492-17621-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.53.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790713339161492-17621-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.17.53.67
--webserver_port=0
--tserver_master_addrs=127.17.53.126:44889
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.53.67
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790713339161492-17621-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:23:51.215097 27321 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:23:51.216964 27321 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:23:51.229815 27327 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:23:51.229791 27326 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:23:51.230014 27329 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:23:51.230610 27321 server_base.cc:1034] running on GCE node
I20260929 20:23:51.231307 27321 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:23:51.232278 27321 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:23:51.233474 27321 hybrid_clock.cc:648] HybridClock initialized: now 1790713431233436 us; error 38 us; skew 500 ppm
I20260929 20:23:51.233884 27321 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 20:23:51.236132 27321 webserver.cc:460] Webserver started at http://127.17.53.67:44825/ using document root <none> and password file <none>
I20260929 20:23:51.236842 27321 fs_manager.cc:359] Metadata directory not provided
I20260929 20:23:51.236958 27321 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:23:51.237283 27321 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:23:51.240137 27321 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790713339161492-17621-0/minicluster-data/ts-2/data/instance:
uuid: "21384d68673a402193ad1923e218bc00"
format_stamp: "Formatted at 2026-09-29 20:23:51 on dist-test-slave-zbnc"
server_key: "e2164f960ff7513f410e96e879914fb8"
server_key_iv: "f2e934e4b451c632ada30d68203d1233"
server_key_version: "encryptionkey@0"
I20260929 20:23:51.240927 27321 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790713339161492-17621-0/minicluster-data/ts-2/wal/instance:
uuid: "21384d68673a402193ad1923e218bc00"
format_stamp: "Formatted at 2026-09-29 20:23:51 on dist-test-slave-zbnc"
server_key: "e2164f960ff7513f410e96e879914fb8"
server_key_iv: "f2e934e4b451c632ada30d68203d1233"
server_key_version: "encryptionkey@0"
I20260929 20:23:51.246174 27321 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260929 20:23:51.249579 27335 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:23:51.250983 27321 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 20:23:51.251164 27321 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790713339161492-17621-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790713339161492-17621-0/minicluster-data/ts-2/wal
uuid: "21384d68673a402193ad1923e218bc00"
format_stamp: "Formatted at 2026-09-29 20:23:51 on dist-test-slave-zbnc"
server_key: "e2164f960ff7513f410e96e879914fb8"
server_key_iv: "f2e934e4b451c632ada30d68203d1233"
server_key_version: "encryptionkey@0"
I20260929 20:23:51.251389 27321 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790713339161492-17621-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790713339161492-17621-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790713339161492-17621-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:23:51.278815 27321 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:23:51.279820 27321 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:23:51.280170 27321 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:23:51.281250 27321 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:23:51.282754 27321 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:23:51.282898 27321 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:23:51.283033 27321 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:23:51.283142 27321 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:23:51.317435 27321 rpc_server.cc:307] RPC server started. Bound to: 127.17.53.67:38607
I20260929 20:23:51.317487 27447 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.53.67:38607 every 8 connection(s)
I20260929 20:23:51.318768 27321 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790713339161492-17621-0/minicluster-data/ts-2/data/info.pb
I20260929 20:23:51.322738 17621 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu as pid 27321
I20260929 20:23:51.322932 17621 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790713339161492-17621-0/minicluster-data/ts-2/wal/instance
I20260929 20:23:51.323494 17621 external_mini_cluster.cc:1428] Setting key c83c65bc25dd7b156b24bcc253bb6592
I20260929 20:23:51.331831 27189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:51.321401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.53.67:49113 (local address 127.17.53.126:44889)
0929 20:23:51.321581 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0929 20:23:51.321585 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:23:51.323260 (+ 1675us) server_negotiation.cc:407] Connection header received
0929 20:23:51.325478 (+ 2218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:23:51.325484 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:23:51.325583 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:23:51.325709 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:23:51.327063 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:51.327782 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:51.329281 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:51.329676 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:51.329774 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:51.331148 (+ 1374us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:23:51.331152 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:23:51.331155 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:23:51.331173 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:23:51.331202 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:23:51.331213 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:23:51.331284 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0929 20:23:51.331521 (+ 237us) server_negotiation.cc:299] Negotiation successful
0929 20:23:51.331595 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 20:23:51.332859 27450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:51.322058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:44889 (local address 127.17.53.67:49113)
0929 20:23:51.323103 (+ 1045us) negotiation.cc:107] Waiting for socket to connect
0929 20:23:51.323146 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0929 20:23:51.325118 (+ 1972us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:23:51.325943 (+ 825us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:23:51.325954 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:23:51.326083 (+ 129us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:23:51.326836 (+ 753us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:23:51.326899 (+ 63us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:23:51.327907 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:23:51.327911 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:23:51.329155 (+ 1244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:23:51.329167 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:23:51.329863 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:23:51.329868 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:23:51.330059 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:51.330784 (+ 725us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 20:23:51.330787 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:23:51.330825 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 20:23:51.330971 (+ 146us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:23:51.331299 (+ 328us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:23:51.331306 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:23:51.331312 (+ 6us) client_negotiation.cc:769] Sending connection context
0929 20:23:51.331454 (+ 142us) client_negotiation.cc:240] Negotiation successful
0929 20:23:51.331758 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":475,"thread_start_us":306,"threads_started":1}
I20260929 20:23:51.334298 27448 heartbeater.cc:347] Connected to a master server at 127.17.53.126:44889
I20260929 20:23:51.334555 27448 heartbeater.cc:464] Registering TS with master...
I20260929 20:23:51.335340 27448 heartbeater.cc:511] Master 127.17.53.126:44889 requested a full tablet report, sending...
I20260929 20:23:51.336675 27001 ts_manager.cc:194] Registered new tserver with Master: 21384d68673a402193ad1923e218bc00 (127.17.53.67:38607)
I20260929 20:23:51.337589 27001 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.17.53.67:49113
I20260929 20:23:51.339411 17621 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260929 20:23:51.346248 27189 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:23:51.344334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41706 (local address 127.17.53.126:44889)
0929 20:23:51.344518 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0929 20:23:51.344522 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:23:51.345053 (+ 531us) server_negotiation.cc:407] Connection header received
0929 20:23:51.345197 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:23:51.345202 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:23:51.345287 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:23:51.345441 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:23:51.345945 (+ 504us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41706: BlockingRecv error: recv got EOF from 127.0.0.1:41706 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 20:23:51.347296 17621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu pid 27060
I20260929 20:23:51.357852 27457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:51.348217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33792 (local address 127.17.53.65:38025)
0929 20:23:51.348780 (+ 563us) server_negotiation.cc:206] Beginning negotiation
0929 20:23:51.348787 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:23:51.348816 (+ 29us) server_negotiation.cc:407] Connection header received
0929 20:23:51.348873 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:23:51.348880 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:23:51.349097 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:23:51.349239 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:23:51.349543 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:51.350645 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:51.352575 (+ 1930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:51.353584 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:51.353735 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:51.355556 (+ 1821us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:23:51.355566 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:23:51.355583 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:23:51.356369 (+ 786us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:23:51.356436 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:23:51.356460 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:23:51.356571 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0929 20:23:51.357302 (+ 731us) server_negotiation.cc:299] Negotiation successful
0929 20:23:51.357442 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":389,"thread_start_us":301,"threads_started":1}
I20260929 20:23:51.359937 27182 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:23:51.463686 27187 heartbeater.cc:503] Master 127.17.53.126:44889 was elected leader, sending a full tablet report...
I20260929 20:23:51.464812 17621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu with pid 27060
W20260929 20:23:51.481710 26985 connection.cc:535] server connection from 127.17.53.65:55019 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260929 20:23:51.482056 17621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu pid 27191
I20260929 20:23:51.489907 27459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:51.482988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45374 (local address 127.17.53.66:45541)
0929 20:23:51.483560 (+ 572us) server_negotiation.cc:206] Beginning negotiation
0929 20:23:51.483568 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 20:23:51.483596 (+ 28us) server_negotiation.cc:407] Connection header received
0929 20:23:51.483664 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:23:51.483671 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:23:51.483886 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:23:51.484020 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:23:51.484375 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:51.485461 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:51.486751 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:51.487393 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:51.487500 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:51.488599 (+ 1099us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:23:51.488608 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:23:51.488622 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:23:51.489215 (+ 593us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:23:51.489261 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:23:51.489283 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:23:51.489372 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0929 20:23:51.489501 (+ 129us) server_negotiation.cc:299] Negotiation successful
0929 20:23:51.489615 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":413,"thread_start_us":287,"threads_started":1}
I20260929 20:23:51.491221 27313 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:23:51.572368 17621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu with pid 27191
I20260929 20:23:51.588486 17621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu pid 27321
I20260929 20:23:51.596604 27461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:51.589521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43304 (local address 127.17.53.67:38607)
0929 20:23:51.590183 (+ 662us) server_negotiation.cc:206] Beginning negotiation
0929 20:23:51.590190 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:23:51.590220 (+ 30us) server_negotiation.cc:407] Connection header received
0929 20:23:51.590282 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:23:51.590289 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:23:51.590508 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:23:51.590643 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:23:51.591114 (+ 471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:51.592130 (+ 1016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:51.593302 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:51.593854 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:51.593949 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:51.595076 (+ 1127us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:23:51.595085 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:23:51.595099 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:23:51.595745 (+ 646us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:23:51.595791 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:23:51.595809 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:23:51.595952 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0929 20:23:51.596118 (+ 166us) server_negotiation.cc:299] Negotiation successful
0929 20:23:51.596248 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":511,"thread_start_us":363,"threads_started":1}
I20260929 20:23:51.597875 27443 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:23:51.680181 17621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu with pid 27321
I20260929 20:23:51.696164 17621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu pid 26970
I20260929 20:23:51.697352 27031 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:23:51.800948 17621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu with pid 26970
2026-09-29T20:23:51Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (2525 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-29T20:23:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T20:23:51Z Disabled control of system clock
I20260929 20:23:51.871836 17621 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
/tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790713339161492-17621-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790713339161492-17621-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790713339161492-17621-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790713339161492-17621-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.53.126:33263
--webserver_interface=127.17.53.126
--webserver_port=0
--builtin_ntp_servers=127.17.53.84:36719
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.53.126:33263
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260929 20:23:52.200306 27467 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:23:52.200609 27467 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:23:52.200788 27467 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:23:52.208135 27467 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:23:52.208251 27467 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:23:52.208328 27467 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:23:52.208421 27467 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:23:52.208509 27467 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:23:52.219266 27467 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.53.84:36719
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790713339161492-17621-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790713339161492-17621-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.53.126:33263
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.53.126:33263
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790713339161492-17621-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.53.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790713339161492-17621-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:23:52.220721 27467 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:23:52.222791 27467 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:23:52.236039 27472 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:23:52.236050 27473 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:23:52.236181 27475 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:23:52.236831 27467 server_base.cc:1034] running on GCE node
I20260929 20:23:52.237637 27467 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:23:52.239195 27467 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:23:52.240435 27467 hybrid_clock.cc:648] HybridClock initialized: now 1790713432240391 us; error 39 us; skew 500 ppm
I20260929 20:23:52.240768 27467 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 20:23:52.243383 27467 webserver.cc:460] Webserver started at http://127.17.53.126:44955/ using document root <none> and password file <none>
I20260929 20:23:52.244115 27467 fs_manager.cc:359] Metadata directory not provided
I20260929 20:23:52.244205 27467 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:23:52.244499 27467 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:23:52.247634 27467 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790713339161492-17621-0/minicluster-data/master-0/data/instance:
uuid: "adb92584e8394ad4a79cd97f04467a1e"
format_stamp: "Formatted at 2026-09-29 20:23:52 on dist-test-slave-zbnc"
server_key: "c31567ebd9d0387ade5260a3f7b4f30b"
server_key_iv: "cfcf7e5d243b6f3e7d2a11f2e0a41d26"
server_key_version: "encryptionkey@0"
I20260929 20:23:52.248472 27467 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790713339161492-17621-0/minicluster-data/master-0/wal/instance:
uuid: "adb92584e8394ad4a79cd97f04467a1e"
format_stamp: "Formatted at 2026-09-29 20:23:52 on dist-test-slave-zbnc"
server_key: "c31567ebd9d0387ade5260a3f7b4f30b"
server_key_iv: "cfcf7e5d243b6f3e7d2a11f2e0a41d26"
server_key_version: "encryptionkey@0"
I20260929 20:23:52.254056 27467 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260929 20:23:52.257468 27481 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:23:52.258846 27467 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260929 20:23:52.259011 27467 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790713339161492-17621-0/minicluster-data/master-0/data,/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790713339161492-17621-0/minicluster-data/master-0/wal
uuid: "adb92584e8394ad4a79cd97f04467a1e"
format_stamp: "Formatted at 2026-09-29 20:23:52 on dist-test-slave-zbnc"
server_key: "c31567ebd9d0387ade5260a3f7b4f30b"
server_key_iv: "cfcf7e5d243b6f3e7d2a11f2e0a41d26"
server_key_version: "encryptionkey@0"
I20260929 20:23:52.259192 27467 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790713339161492-17621-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790713339161492-17621-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790713339161492-17621-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:23:52.264904 27467 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:23:52.265630 27467 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:23:52.265873 27467 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:23:52.284394 27467 rpc_server.cc:307] RPC server started. Bound to: 127.17.53.126:33263
I20260929 20:23:52.284426 27532 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.53.126:33263 every 8 connection(s)
I20260929 20:23:52.285737 27467 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790713339161492-17621-0/minicluster-data/master-0/data/info.pb
I20260929 20:23:52.289544 27533 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:23:52.293485 17621 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu as pid 27467
I20260929 20:23:52.293689 17621 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790713339161492-17621-0/minicluster-data/master-0/wal/instance
I20260929 20:23:52.294322 17621 external_mini_cluster.cc:1428] Setting key e93f4dc1f3fa1250f4784a89dd9ed921
I20260929 20:23:52.300263 27533 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P adb92584e8394ad4a79cd97f04467a1e: Bootstrap starting.
I20260929 20:23:52.302683 27536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:52.296339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45964 (local address 127.17.53.126:33263)
0929 20:23:52.297409 (+ 1070us) server_negotiation.cc:206] Beginning negotiation
0929 20:23:52.297422 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 20:23:52.297464 (+ 42us) server_negotiation.cc:407] Connection header received
0929 20:23:52.299195 (+ 1731us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:23:52.299217 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:23:52.299623 (+ 406us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:23:52.299934 (+ 311us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:23:52.301049 (+ 1115us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:23:52.301056 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:23:52.301069 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:23:52.301084 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:23:52.301151 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:23:52.301167 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:23:52.301390 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0929 20:23:52.301523 (+ 133us) server_negotiation.cc:299] Negotiation successful
0929 20:23:52.301809 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":629,"thread_start_us":361,"threads_started":1}
I20260929 20:23:52.303552 27533 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P adb92584e8394ad4a79cd97f04467a1e: Neither blocks nor log segments found. Creating new log.
I20260929 20:23:52.304576 27533 log.cc:824] T 00000000000000000000000000000000 P adb92584e8394ad4a79cd97f04467a1e: Log is configured to *not* fsync() on all Append() calls
I20260929 20:23:52.307325 27533 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P adb92584e8394ad4a79cd97f04467a1e: No bootstrap required, opened a new log
I20260929 20:23:52.314390 27533 raft_consensus.cc:348] T 00000000000000000000000000000000 P adb92584e8394ad4a79cd97f04467a1e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "adb92584e8394ad4a79cd97f04467a1e" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 33263 } }
I20260929 20:23:52.314790 27533 raft_consensus.cc:374] T 00000000000000000000000000000000 P adb92584e8394ad4a79cd97f04467a1e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:23:52.314908 27533 raft_consensus.cc:729] T 00000000000000000000000000000000 P adb92584e8394ad4a79cd97f04467a1e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: adb92584e8394ad4a79cd97f04467a1e, State: Initialized, Role: FOLLOWER
I20260929 20:23:52.315429 27533 consensus_queue.cc:260] T 00000000000000000000000000000000 P adb92584e8394ad4a79cd97f04467a1e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "adb92584e8394ad4a79cd97f04467a1e" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 33263 } }
I20260929 20:23:52.315660 27533 raft_consensus.cc:388] T 00000000000000000000000000000000 P adb92584e8394ad4a79cd97f04467a1e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 20:23:52.315788 27533 raft_consensus.cc:482] T 00000000000000000000000000000000 P adb92584e8394ad4a79cd97f04467a1e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 20:23:52.315960 27533 raft_consensus.cc:3037] T 00000000000000000000000000000000 P adb92584e8394ad4a79cd97f04467a1e [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:23:52.318045 27533 raft_consensus.cc:504] T 00000000000000000000000000000000 P adb92584e8394ad4a79cd97f04467a1e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "adb92584e8394ad4a79cd97f04467a1e" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 33263 } }
I20260929 20:23:52.318529 27533 leader_election.cc:302] T 00000000000000000000000000000000 P adb92584e8394ad4a79cd97f04467a1e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: adb92584e8394ad4a79cd97f04467a1e; no voters:
I20260929 20:23:52.319172 27533 leader_election.cc:288] T 00000000000000000000000000000000 P adb92584e8394ad4a79cd97f04467a1e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 20:23:52.319300 27538 raft_consensus.cc:2781] T 00000000000000000000000000000000 P adb92584e8394ad4a79cd97f04467a1e [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:23:52.320055 27538 raft_consensus.cc:686] T 00000000000000000000000000000000 P adb92584e8394ad4a79cd97f04467a1e [term 1 LEADER]: Becoming Leader. State: Replica: adb92584e8394ad4a79cd97f04467a1e, State: Running, Role: LEADER
I20260929 20:23:52.320611 27538 consensus_queue.cc:237] T 00000000000000000000000000000000 P adb92584e8394ad4a79cd97f04467a1e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "adb92584e8394ad4a79cd97f04467a1e" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 33263 } }
I20260929 20:23:52.321256 27533 sys_catalog.cc:564] T 00000000000000000000000000000000 P adb92584e8394ad4a79cd97f04467a1e [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:23:52.323904 27540 sys_catalog.cc:455] T 00000000000000000000000000000000 P adb92584e8394ad4a79cd97f04467a1e [sys.catalog]: SysCatalogTable state changed. Reason: New leader adb92584e8394ad4a79cd97f04467a1e. Latest consensus state: current_term: 1 leader_uuid: "adb92584e8394ad4a79cd97f04467a1e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "adb92584e8394ad4a79cd97f04467a1e" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 33263 } } }
I20260929 20:23:52.323889 27539 sys_catalog.cc:455] T 00000000000000000000000000000000 P adb92584e8394ad4a79cd97f04467a1e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "adb92584e8394ad4a79cd97f04467a1e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "adb92584e8394ad4a79cd97f04467a1e" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 33263 } } }
I20260929 20:23:52.324306 27540 sys_catalog.cc:458] T 00000000000000000000000000000000 P adb92584e8394ad4a79cd97f04467a1e [sys.catalog]: This master's current role is: LEADER
I20260929 20:23:52.324369 27539 sys_catalog.cc:458] T 00000000000000000000000000000000 P adb92584e8394ad4a79cd97f04467a1e [sys.catalog]: This master's current role is: LEADER
I20260929 20:23:52.327636 27545 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:23:52.333084 27545 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:23:52.341806 27545 catalog_manager.cc:1347] Generated new cluster ID: 2b8d05e9bf5d4297b7988d9aa169d756
I20260929 20:23:52.341915 27545 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:23:52.369387 27545 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 20:23:52.369827 27545 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:23:52.379882 27545 catalog_manager.cc:5868] T 00000000000000000000000000000000 P adb92584e8394ad4a79cd97f04467a1e: Generated new TSK 0
I20260929 20:23:52.380748 27545 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 20:23:52.399086 17621 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
/tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790713339161492-17621-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790713339161492-17621-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790713339161492-17621-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790713339161492-17621-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.53.65:0
--local_ip_for_outbound_sockets=127.17.53.65
--webserver_interface=127.17.53.65
--webserver_port=0
--tserver_master_addrs=127.17.53.126:33263
--builtin_ntp_servers=127.17.53.84:36719
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260929 20:23:52.747259 27557 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:23:52.747545 27557 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:23:52.747713 27557 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:23:52.755038 27557 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:23:52.755206 27557 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:23:52.755406 27557 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.53.65
I20260929 20:23:52.765003 27557 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.53.84:36719
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790713339161492-17621-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790713339161492-17621-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.53.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790713339161492-17621-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.17.53.65
--webserver_port=0
--tserver_master_addrs=127.17.53.126:33263
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.53.65
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790713339161492-17621-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:23:52.766409 27557 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:23:52.768204 27557 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:23:52.781678 27562 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:23:52.781700 27563 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:23:52.781976 27565 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:23:52.782455 27557 server_base.cc:1034] running on GCE node
I20260929 20:23:52.783273 27557 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:23:52.784307 27557 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:23:52.785547 27557 hybrid_clock.cc:648] HybridClock initialized: now 1790713432785496 us; error 44 us; skew 500 ppm
I20260929 20:23:52.785868 27557 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 20:23:52.788353 27557 webserver.cc:460] Webserver started at http://127.17.53.65:39127/ using document root <none> and password file <none>
I20260929 20:23:52.789098 27557 fs_manager.cc:359] Metadata directory not provided
I20260929 20:23:52.789235 27557 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:23:52.789568 27557 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:23:52.792526 27557 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790713339161492-17621-0/minicluster-data/ts-0/data/instance:
uuid: "2a5d60449bfb4a1d814b37fc9a56a377"
format_stamp: "Formatted at 2026-09-29 20:23:52 on dist-test-slave-zbnc"
server_key: "a69f67d1c9b6bd3b12b0f0e48c308a16"
server_key_iv: "3c0cde519c161a65ca006c736eb515eb"
server_key_version: "encryptionkey@0"
I20260929 20:23:52.793370 27557 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790713339161492-17621-0/minicluster-data/ts-0/wal/instance:
uuid: "2a5d60449bfb4a1d814b37fc9a56a377"
format_stamp: "Formatted at 2026-09-29 20:23:52 on dist-test-slave-zbnc"
server_key: "a69f67d1c9b6bd3b12b0f0e48c308a16"
server_key_iv: "3c0cde519c161a65ca006c736eb515eb"
server_key_version: "encryptionkey@0"
I20260929 20:23:52.798709 27557 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260929 20:23:52.802336 27571 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:23:52.803893 27557 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 20:23:52.804070 27557 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790713339161492-17621-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790713339161492-17621-0/minicluster-data/ts-0/wal
uuid: "2a5d60449bfb4a1d814b37fc9a56a377"
format_stamp: "Formatted at 2026-09-29 20:23:52 on dist-test-slave-zbnc"
server_key: "a69f67d1c9b6bd3b12b0f0e48c308a16"
server_key_iv: "3c0cde519c161a65ca006c736eb515eb"
server_key_version: "encryptionkey@0"
I20260929 20:23:52.804312 27557 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790713339161492-17621-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790713339161492-17621-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790713339161492-17621-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:23:52.810195 27557 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:23:52.811066 27557 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:23:52.811336 27557 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:23:52.812294 27557 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:23:52.813760 27557 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:23:52.813881 27557 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:23:52.814010 27557 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:23:52.814096 27557 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:23:52.851413 27557 rpc_server.cc:307] RPC server started. Bound to: 127.17.53.65:39541
I20260929 20:23:52.851457 27683 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.53.65:39541 every 8 connection(s)
I20260929 20:23:52.852759 27557 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790713339161492-17621-0/minicluster-data/ts-0/data/info.pb
I20260929 20:23:52.857473 17621 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu as pid 27557
I20260929 20:23:52.857740 17621 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790713339161492-17621-0/minicluster-data/ts-0/wal/instance
I20260929 20:23:52.858760 17621 external_mini_cluster.cc:1428] Setting key 8cb54dfbe39c9711389adacea61aa03c
I20260929 20:23:52.862351 17621 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
/tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790713339161492-17621-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790713339161492-17621-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790713339161492-17621-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790713339161492-17621-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.53.66:0
--local_ip_for_outbound_sockets=127.17.53.66
--webserver_interface=127.17.53.66
--webserver_port=0
--tserver_master_addrs=127.17.53.126:33263
--builtin_ntp_servers=127.17.53.84:36719
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260929 20:23:52.863489 27686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:52.855751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.53.65:34339 (local address 127.17.53.126:33263)
0929 20:23:52.856251 (+ 500us) server_negotiation.cc:206] Beginning negotiation
0929 20:23:52.856256 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:23:52.857856 (+ 1600us) server_negotiation.cc:407] Connection header received
0929 20:23:52.860627 (+ 2771us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:23:52.860634 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:23:52.860693 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:23:52.860824 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:23:52.862642 (+ 1818us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:23:52.862650 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:23:52.862656 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:23:52.862677 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:23:52.862729 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:23:52.862742 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:23:52.862842 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0929 20:23:52.863157 (+ 315us) server_negotiation.cc:299] Negotiation successful
0929 20:23:52.863207 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":385,"thread_start_us":260,"threads_started":1}
I20260929 20:23:52.865043 27687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:52.856374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:33263 (local address 127.17.53.65:34339)
0929 20:23:52.857643 (+ 1269us) negotiation.cc:107] Waiting for socket to connect
0929 20:23:52.857713 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0929 20:23:52.860304 (+ 2591us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:23:52.861182 (+ 878us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:23:52.861198 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:23:52.861356 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:23:52.862157 (+ 801us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 20:23:52.862161 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:23:52.862221 (+ 60us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 20:23:52.862358 (+ 137us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:23:52.862886 (+ 528us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:23:52.862902 (+ 16us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:23:52.862914 (+ 12us) client_negotiation.cc:769] Sending connection context
0929 20:23:52.863269 (+ 355us) client_negotiation.cc:240] Negotiation successful
0929 20:23:52.863592 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":588,"thread_start_us":303,"threads_started":1}
I20260929 20:23:52.866631 27684 heartbeater.cc:347] Connected to a master server at 127.17.53.126:33263
I20260929 20:23:52.867053 27684 heartbeater.cc:464] Registering TS with master...
I20260929 20:23:52.867892 27684 heartbeater.cc:511] Master 127.17.53.126:33263 requested a full tablet report, sending...
I20260929 20:23:52.870278 27498 ts_manager.cc:194] Registered new tserver with Master: 2a5d60449bfb4a1d814b37fc9a56a377 (127.17.53.65:39541)
W20260929 20:23:53.199661 27688 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:23:53.199939 27688 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:23:53.200028 27688 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:23:53.207288 27688 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:23:53.207392 27688 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:23:53.207568 27688 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.53.66
I20260929 20:23:53.216719 27688 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.53.84:36719
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790713339161492-17621-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790713339161492-17621-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.53.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790713339161492-17621-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.17.53.66
--webserver_port=0
--tserver_master_addrs=127.17.53.126:33263
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.53.66
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790713339161492-17621-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:23:53.217989 27688 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:23:53.219784 27688 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:23:53.233544 27693 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:23:53.233654 27696 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:23:53.234061 27694 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:23:53.234586 27688 server_base.cc:1034] running on GCE node
I20260929 20:23:53.235342 27688 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:23:53.236326 27688 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:23:53.237602 27688 hybrid_clock.cc:648] HybridClock initialized: now 1790713433237529 us; error 50 us; skew 500 ppm
I20260929 20:23:53.237999 27688 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 20:23:53.240459 27688 webserver.cc:460] Webserver started at http://127.17.53.66:43973/ using document root <none> and password file <none>
I20260929 20:23:53.241209 27688 fs_manager.cc:359] Metadata directory not provided
I20260929 20:23:53.241345 27688 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:23:53.241672 27688 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:23:53.244519 27688 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790713339161492-17621-0/minicluster-data/ts-1/data/instance:
uuid: "7349abc3917d4479a4377d31f55bd291"
format_stamp: "Formatted at 2026-09-29 20:23:53 on dist-test-slave-zbnc"
server_key: "f23e5ab9a8f1f02a4c1ba32737c296a2"
server_key_iv: "3e7ddf3009bc6e02e9d37d938f03978e"
server_key_version: "encryptionkey@0"
I20260929 20:23:53.245337 27688 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790713339161492-17621-0/minicluster-data/ts-1/wal/instance:
uuid: "7349abc3917d4479a4377d31f55bd291"
format_stamp: "Formatted at 2026-09-29 20:23:53 on dist-test-slave-zbnc"
server_key: "f23e5ab9a8f1f02a4c1ba32737c296a2"
server_key_iv: "3e7ddf3009bc6e02e9d37d938f03978e"
server_key_version: "encryptionkey@0"
I20260929 20:23:53.250313 27688 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.003s
I20260929 20:23:53.253669 27702 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:23:53.255005 27688 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260929 20:23:53.255177 27688 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790713339161492-17621-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790713339161492-17621-0/minicluster-data/ts-1/wal
uuid: "7349abc3917d4479a4377d31f55bd291"
format_stamp: "Formatted at 2026-09-29 20:23:53 on dist-test-slave-zbnc"
server_key: "f23e5ab9a8f1f02a4c1ba32737c296a2"
server_key_iv: "3e7ddf3009bc6e02e9d37d938f03978e"
server_key_version: "encryptionkey@0"
I20260929 20:23:53.255391 27688 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790713339161492-17621-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790713339161492-17621-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790713339161492-17621-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:23:53.260842 27688 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:23:53.261595 27688 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:23:53.261873 27688 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:23:53.262745 27688 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:23:53.264289 27688 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:23:53.264375 27688 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:23:53.264523 27688 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:23:53.264614 27688 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:23:53.311239 27688 rpc_server.cc:307] RPC server started. Bound to: 127.17.53.66:44283
I20260929 20:23:53.311288 27815 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.53.66:44283 every 8 connection(s)
I20260929 20:23:53.312592 27688 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790713339161492-17621-0/minicluster-data/ts-1/data/info.pb
I20260929 20:23:53.317615 17621 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu as pid 27688
I20260929 20:23:53.317790 17621 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790713339161492-17621-0/minicluster-data/ts-1/wal/instance
I20260929 20:23:53.318253 17621 external_mini_cluster.cc:1428] Setting key d814709382dbda006631890d1de8bc88
I20260929 20:23:53.321228 27686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:53.315234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.53.66:60427 (local address 127.17.53.126:33263)
0929 20:23:53.315454 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0929 20:23:53.315459 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:23:53.316859 (+ 1400us) server_negotiation.cc:407] Connection header received
0929 20:23:53.318801 (+ 1942us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:23:53.318805 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:23:53.318947 (+ 142us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:23:53.319105 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:23:53.320562 (+ 1457us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:23:53.320566 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:23:53.320569 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:23:53.320583 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:23:53.320624 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:23:53.320635 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:23:53.320721 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0929 20:23:53.320944 (+ 223us) server_negotiation.cc:299] Negotiation successful
0929 20:23:53.321025 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260929 20:23:53.321430 17621 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
/tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790713339161492-17621-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790713339161492-17621-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790713339161492-17621-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790713339161492-17621-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.53.67:0
--local_ip_for_outbound_sockets=127.17.53.67
--webserver_interface=127.17.53.67
--webserver_port=0
--tserver_master_addrs=127.17.53.126:33263
--builtin_ntp_servers=127.17.53.84:36719
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260929 20:23:53.322507 27818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:53.315778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:33263 (local address 127.17.53.66:60427)
0929 20:23:53.316680 (+ 902us) negotiation.cc:107] Waiting for socket to connect
0929 20:23:53.316715 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0929 20:23:53.318521 (+ 1806us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:23:53.319345 (+ 824us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:23:53.319356 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:23:53.319474 (+ 118us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:23:53.320207 (+ 733us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 20:23:53.320210 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:23:53.320280 (+ 70us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 20:23:53.320395 (+ 115us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:23:53.320734 (+ 339us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:23:53.320744 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:23:53.320751 (+ 7us) client_negotiation.cc:769] Sending connection context
0929 20:23:53.321035 (+ 284us) client_negotiation.cc:240] Negotiation successful
0929 20:23:53.321290 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":289,"threads_started":1}
I20260929 20:23:53.323835 27816 heartbeater.cc:347] Connected to a master server at 127.17.53.126:33263
I20260929 20:23:53.324157 27816 heartbeater.cc:464] Registering TS with master...
I20260929 20:23:53.324683 27816 heartbeater.cc:511] Master 127.17.53.126:33263 requested a full tablet report, sending...
I20260929 20:23:53.325982 27498 ts_manager.cc:194] Registered new tserver with Master: 7349abc3917d4479a4377d31f55bd291 (127.17.53.66:44283)
W20260929 20:23:53.651887 27819 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:23:53.652196 27819 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:23:53.652349 27819 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:23:53.659937 27819 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:23:53.660080 27819 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:23:53.660269 27819 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.53.67
I20260929 20:23:53.669620 27819 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.53.84:36719
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790713339161492-17621-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790713339161492-17621-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.53.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790713339161492-17621-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.17.53.67
--webserver_port=0
--tserver_master_addrs=127.17.53.126:33263
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.53.67
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790713339161492-17621-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:23:53.670957 27819 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:23:53.672758 27819 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:23:53.686105 27824 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:23:53.686151 27825 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:23:53.686076 27827 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:23:53.687016 27819 server_base.cc:1034] running on GCE node
I20260929 20:23:53.687618 27819 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:23:53.688557 27819 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:23:53.689790 27819 hybrid_clock.cc:648] HybridClock initialized: now 1790713433689739 us; error 48 us; skew 500 ppm
I20260929 20:23:53.690120 27819 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 20:23:53.692548 27819 webserver.cc:460] Webserver started at http://127.17.53.67:41633/ using document root <none> and password file <none>
I20260929 20:23:53.693349 27819 fs_manager.cc:359] Metadata directory not provided
I20260929 20:23:53.693468 27819 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:23:53.693804 27819 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:23:53.696753 27819 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790713339161492-17621-0/minicluster-data/ts-2/data/instance:
uuid: "36b483c6776440a9ab415394ef24e7c2"
format_stamp: "Formatted at 2026-09-29 20:23:53 on dist-test-slave-zbnc"
server_key: "054f96d77c4a3789793d2f3c7cb581ed"
server_key_iv: "1f0165374b643956767212b4436ef9b9"
server_key_version: "encryptionkey@0"
I20260929 20:23:53.697566 27819 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790713339161492-17621-0/minicluster-data/ts-2/wal/instance:
uuid: "36b483c6776440a9ab415394ef24e7c2"
format_stamp: "Formatted at 2026-09-29 20:23:53 on dist-test-slave-zbnc"
server_key: "054f96d77c4a3789793d2f3c7cb581ed"
server_key_iv: "1f0165374b643956767212b4436ef9b9"
server_key_version: "encryptionkey@0"
I20260929 20:23:53.702756 27819 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260929 20:23:53.706355 27833 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:23:53.707841 27819 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 20:23:53.708012 27819 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790713339161492-17621-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790713339161492-17621-0/minicluster-data/ts-2/wal
uuid: "36b483c6776440a9ab415394ef24e7c2"
format_stamp: "Formatted at 2026-09-29 20:23:53 on dist-test-slave-zbnc"
server_key: "054f96d77c4a3789793d2f3c7cb581ed"
server_key_iv: "1f0165374b643956767212b4436ef9b9"
server_key_version: "encryptionkey@0"
I20260929 20:23:53.708246 27819 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790713339161492-17621-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790713339161492-17621-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790713339161492-17621-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:23:53.713979 27819 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:23:53.714763 27819 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:23:53.715054 27819 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:23:53.715924 27819 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:23:53.717257 27819 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:23:53.717376 27819 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:23:53.717519 27819 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:23:53.717609 27819 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:23:53.751452 27819 rpc_server.cc:307] RPC server started. Bound to: 127.17.53.67:45385
I20260929 20:23:53.751503 27946 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.53.67:45385 every 8 connection(s)
I20260929 20:23:53.752836 27819 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790713339161492-17621-0/minicluster-data/ts-2/data/info.pb
I20260929 20:23:53.754434 17621 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu as pid 27819
I20260929 20:23:53.754645 17621 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790713339161492-17621-0/minicluster-data/ts-2/wal/instance
I20260929 20:23:53.755252 17621 external_mini_cluster.cc:1428] Setting key 2f65bcfd56601da353170516569fabc7
I20260929 20:23:53.761646 27686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:53.755444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.53.67:47319 (local address 127.17.53.126:33263)
0929 20:23:53.755631 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0929 20:23:53.755635 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:23:53.757496 (+ 1861us) server_negotiation.cc:407] Connection header received
0929 20:23:53.759540 (+ 2044us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:23:53.759544 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:23:53.759625 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:23:53.759729 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:23:53.761029 (+ 1300us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:23:53.761033 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:23:53.761037 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:23:53.761051 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:23:53.761082 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:23:53.761093 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:23:53.761180 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0929 20:23:53.761349 (+ 169us) server_negotiation.cc:299] Negotiation successful
0929 20:23:53.761396 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 20:23:53.762627 27949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:53.756195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:33263 (local address 127.17.53.67:47319)
0929 20:23:53.757307 (+ 1112us) negotiation.cc:107] Waiting for socket to connect
0929 20:23:53.757369 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0929 20:23:53.759240 (+ 1871us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:23:53.759995 (+ 755us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:23:53.760006 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:23:53.760125 (+ 119us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:23:53.760765 (+ 640us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 20:23:53.760767 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:23:53.760803 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 20:23:53.760884 (+ 81us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:23:53.761176 (+ 292us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:23:53.761183 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:23:53.761188 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 20:23:53.761398 (+ 210us) client_negotiation.cc:240] Negotiation successful
0929 20:23:53.761592 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":560,"thread_start_us":357,"threads_started":1}
I20260929 20:23:53.764113 27947 heartbeater.cc:347] Connected to a master server at 127.17.53.126:33263
I20260929 20:23:53.764508 27947 heartbeater.cc:464] Registering TS with master...
I20260929 20:23:53.765115 27947 heartbeater.cc:511] Master 127.17.53.126:33263 requested a full tablet report, sending...
I20260929 20:23:53.766518 27498 ts_manager.cc:194] Registered new tserver with Master: 36b483c6776440a9ab415394ef24e7c2 (127.17.53.67:45385)
I20260929 20:23:53.770581 17621 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260929 20:23:53.777333 27686 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:23:53.775842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45976 (local address 127.17.53.126:33263)
0929 20:23:53.776035 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0929 20:23:53.776044 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 20:23:53.776538 (+ 494us) server_negotiation.cc:407] Connection header received
0929 20:23:53.776677 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:23:53.776680 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:23:53.776729 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:23:53.776808 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:23:53.777094 (+ 286us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45976: BlockingRecv error: recv got EOF from 127.0.0.1:45976 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
I20260929 20:23:53.777724 17621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu pid 27557
I20260929 20:23:53.781078 27956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:53.778663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44608 (local address 127.17.53.65:39541)
0929 20:23:53.779207 (+ 544us) server_negotiation.cc:206] Beginning negotiation
0929 20:23:53.779213 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:23:53.779238 (+ 25us) server_negotiation.cc:407] Connection header received
0929 20:23:53.779295 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:23:53.779301 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:23:53.779510 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:23:53.779633 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:23:53.780506 (+ 873us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:23:53.780512 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:23:53.780524 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:23:53.780540 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:23:53.780577 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:23:53.780591 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:23:53.780693 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0929 20:23:53.780773 (+ 80us) server_negotiation.cc:299] Negotiation successful
0929 20:23:53.780847 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":393,"thread_start_us":275,"threads_started":1}
I20260929 20:23:53.782239 27679 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:23:53.863086 17621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu with pid 27557
I20260929 20:23:53.879446 17621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu pid 27688
I20260929 20:23:53.883163 27958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:53.880410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41782 (local address 127.17.53.66:44283)
0929 20:23:53.880900 (+ 490us) server_negotiation.cc:206] Beginning negotiation
0929 20:23:53.880907 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:23:53.880933 (+ 26us) server_negotiation.cc:407] Connection header received
0929 20:23:53.880995 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:23:53.881001 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:23:53.881224 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:23:53.881396 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:23:53.882392 (+ 996us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:23:53.882398 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:23:53.882411 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:23:53.882423 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:23:53.882490 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:23:53.882504 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:23:53.882649 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0929 20:23:53.882788 (+ 139us) server_negotiation.cc:299] Negotiation successful
0929 20:23:53.882911 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":374,"thread_start_us":261,"threads_started":1}
I20260929 20:23:53.884328 27811 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:23:53.966325 17621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu with pid 27688
I20260929 20:23:53.982636 17621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu pid 27819
I20260929 20:23:53.986573 27960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:53.983684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51878 (local address 127.17.53.67:45385)
0929 20:23:53.984404 (+ 720us) server_negotiation.cc:206] Beginning negotiation
0929 20:23:53.984419 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 20:23:53.984508 (+ 89us) server_negotiation.cc:407] Connection header received
0929 20:23:53.984573 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:23:53.984579 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:23:53.984800 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:23:53.984929 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:23:53.985831 (+ 902us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:23:53.985837 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:23:53.985848 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:23:53.985863 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:23:53.985901 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:23:53.985914 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:23:53.986047 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0929 20:23:53.986097 (+ 50us) server_negotiation.cc:299] Negotiation successful
0929 20:23:53.986217 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":573,"thread_start_us":389,"threads_started":1}
I20260929 20:23:53.987941 27942 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:23:54.069639 17621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu with pid 27819
I20260929 20:23:54.085713 17621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu pid 27467
I20260929 20:23:54.087149 27528 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:23:54.187335 17621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu with pid 27467
2026-09-29T20:23:54Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (2386 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790713339161492-17621-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 20:23:54 dist-test-slave-zbnc krb5kdc[27965](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 20:23:54 dist-test-slave-zbnc krb5kdc[27965](info): set up 2 sockets
Sep 29 20:23:54 dist-test-slave-zbnc krb5kdc[27965](info): commencing operation
krb5kdc: starting...
W20260929 20:23:55.297150 17621 mini_kdc.cc:121] Time spent starting KDC: real 1.069s user 0.002s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790713339161492-17621-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790713339161492-17621-0/krb5kdc/test-user.keytab.
Sep 29 20:23:55 dist-test-slave-zbnc krb5kdc[27965](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713435, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T20:23:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T20:23:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.17.53.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.53.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.53.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.126.keytab.
Entry for principal kudu/127.17.53.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.126.keytab.
I20260929 20:23:55.524474 17621 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
/tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790713339161492-17621-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790713339161492-17621-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790713339161492-17621-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790713339161492-17621-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.53.126:37541
--webserver_interface=127.17.53.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.126.keytab
--principal=kudu/127.17.53.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.53.84:41219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.53.126:37541
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790713339161492-17621-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790713339161492-17621-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790713339161492-17621-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:23:55.852808 27978 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:23:55.853103 27978 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:23:55.853211 27978 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:23:55.860344 27978 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:23:55.860461 27978 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:23:55.860509 27978 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:23:55.860566 27978 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:23:55.860618 27978 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:23:55.870285 27978 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.53.84:41219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790713339161492-17621-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790713339161492-17621-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.53.126:37541
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.126.keytab
--principal=kudu/127.17.53.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.53.126:37541
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790713339161492-17621-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.53.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790713339161492-17621-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:23:55.871584 27978 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:23:55.872990 27978 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:23:55.884742 27983 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:23:55.884666 27984 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:23:55.884711 27986 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:23:55.885658 27978 server_base.cc:1034] running on GCE node
I20260929 20:23:55.886397 27978 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:23:55.887843 27978 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:23:55.889073 27978 hybrid_clock.cc:648] HybridClock initialized: now 1790713435889028 us; error 78 us; skew 500 ppm
I20260929 20:23:55.889415 27978 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:23:55 dist-test-slave-zbnc krb5kdc[27965](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713435, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.53.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:23:55.897897 27978 init.cc:377] Logged in from keytab as kudu/127.17.53.126@KRBTEST.COM (short username kudu)
I20260929 20:23:55.899569 27978 webserver.cc:460] Webserver started at http://127.17.53.126:46033/ using document root <none> and password file <none>
I20260929 20:23:55.900417 27978 fs_manager.cc:359] Metadata directory not provided
I20260929 20:23:55.900564 27978 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:23:55.901022 27978 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:23:55.904354 27978 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790713339161492-17621-0/minicluster-data/master-0/data/instance:
uuid: "55e404fbcffa45f392dc529f09a895d8"
format_stamp: "Formatted at 2026-09-29 20:23:55 on dist-test-slave-zbnc"
server_key: "2bbe290ae346659c2f641e5d4b839f6a"
server_key_iv: "1fb6a7a5f3deeaf46a9152f318b1ac8a"
server_key_version: "encryptionkey@0"
I20260929 20:23:55.905169 27978 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790713339161492-17621-0/minicluster-data/master-0/wal/instance:
uuid: "55e404fbcffa45f392dc529f09a895d8"
format_stamp: "Formatted at 2026-09-29 20:23:55 on dist-test-slave-zbnc"
server_key: "2bbe290ae346659c2f641e5d4b839f6a"
server_key_iv: "1fb6a7a5f3deeaf46a9152f318b1ac8a"
server_key_version: "encryptionkey@0"
I20260929 20:23:55.910415 27978 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260929 20:23:55.913965 27993 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:23:55.915336 27978 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 20:23:55.915521 27978 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790713339161492-17621-0/minicluster-data/master-0/data,/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790713339161492-17621-0/minicluster-data/master-0/wal
uuid: "55e404fbcffa45f392dc529f09a895d8"
format_stamp: "Formatted at 2026-09-29 20:23:55 on dist-test-slave-zbnc"
server_key: "2bbe290ae346659c2f641e5d4b839f6a"
server_key_iv: "1fb6a7a5f3deeaf46a9152f318b1ac8a"
server_key_version: "encryptionkey@0"
I20260929 20:23:55.915719 27978 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790713339161492-17621-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790713339161492-17621-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790713339161492-17621-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:23:55.930147 27978 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:23:55.934549 27978 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:23:55.934849 27978 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:23:55.953611 27978 rpc_server.cc:307] RPC server started. Bound to: 127.17.53.126:37541
I20260929 20:23:55.953631 28044 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.53.126:37541 every 8 connection(s)
I20260929 20:23:55.955070 27978 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790713339161492-17621-0/minicluster-data/master-0/data/info.pb
I20260929 20:23:55.957167 17621 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu as pid 27978
I20260929 20:23:55.957396 17621 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790713339161492-17621-0/minicluster-data/master-0/wal/instance
I20260929 20:23:55.958212 17621 external_mini_cluster.cc:1428] Setting key 01940320c96c4fb6054e347761a9b540
I20260929 20:23:55.959789 28045 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:23:55.971405 28045 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 55e404fbcffa45f392dc529f09a895d8: Bootstrap starting.
Sep 29 20:23:55 dist-test-slave-zbnc krb5kdc[27965](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713435, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.53.126@KRBTEST.COM
I20260929 20:23:55.975210 28045 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 55e404fbcffa45f392dc529f09a895d8: Neither blocks nor log segments found. Creating new log.
I20260929 20:23:55.976307 28045 log.cc:824] T 00000000000000000000000000000000 P 55e404fbcffa45f392dc529f09a895d8: Log is configured to *not* fsync() on all Append() calls
I20260929 20:23:55.979700 28045 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 55e404fbcffa45f392dc529f09a895d8: No bootstrap required, opened a new log
I20260929 20:23:55.980662 28048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:55.960416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40900 (local address 127.17.53.126:37541)
0929 20:23:55.961170 (+ 754us) server_negotiation.cc:206] Beginning negotiation
0929 20:23:55.961182 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 20:23:55.961223 (+ 41us) server_negotiation.cc:407] Connection header received
0929 20:23:55.962756 (+ 1533us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:23:55.962775 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:23:55.963246 (+ 471us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:23:55.963655 (+ 409us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:23:55.965137 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:55.966430 (+ 1293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:55.967747 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:55.968485 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:55.968711 (+ 226us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:55.973154 (+ 4443us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:23:55.973167 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:23:55.973182 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:23:55.973219 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:23:55.976519 (+ 3300us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:23:55.977321 (+ 802us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:23:55.977330 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:23:55.977342 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:23:55.977606 (+ 264us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:23:55.978000 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:23:55.978003 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:23:55.978005 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:23:55.978545 (+ 540us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:23:55.978764 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0929 20:23:55.979248 (+ 484us) server_negotiation.cc:299] Negotiation successful
0929 20:23:55.979648 (+ 400us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":555,"thread_start_us":350,"threads_started":1}
I20260929 20:23:55.987725 28045 raft_consensus.cc:348] T 00000000000000000000000000000000 P 55e404fbcffa45f392dc529f09a895d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55e404fbcffa45f392dc529f09a895d8" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 37541 } }
I20260929 20:23:55.988080 28045 raft_consensus.cc:374] T 00000000000000000000000000000000 P 55e404fbcffa45f392dc529f09a895d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:23:55.988201 28045 raft_consensus.cc:729] T 00000000000000000000000000000000 P 55e404fbcffa45f392dc529f09a895d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 55e404fbcffa45f392dc529f09a895d8, State: Initialized, Role: FOLLOWER
I20260929 20:23:55.988742 28045 consensus_queue.cc:260] T 00000000000000000000000000000000 P 55e404fbcffa45f392dc529f09a895d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55e404fbcffa45f392dc529f09a895d8" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 37541 } }
I20260929 20:23:55.988967 28045 raft_consensus.cc:388] T 00000000000000000000000000000000 P 55e404fbcffa45f392dc529f09a895d8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 20:23:55.989101 28045 raft_consensus.cc:482] T 00000000000000000000000000000000 P 55e404fbcffa45f392dc529f09a895d8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 20:23:55.989265 28045 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 55e404fbcffa45f392dc529f09a895d8 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:23:55.991478 28045 raft_consensus.cc:504] T 00000000000000000000000000000000 P 55e404fbcffa45f392dc529f09a895d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55e404fbcffa45f392dc529f09a895d8" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 37541 } }
I20260929 20:23:55.992043 28045 leader_election.cc:302] T 00000000000000000000000000000000 P 55e404fbcffa45f392dc529f09a895d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 55e404fbcffa45f392dc529f09a895d8; no voters:
I20260929 20:23:55.992641 28045 leader_election.cc:288] T 00000000000000000000000000000000 P 55e404fbcffa45f392dc529f09a895d8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 20:23:55.992771 28050 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 55e404fbcffa45f392dc529f09a895d8 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:23:55.994231 28050 raft_consensus.cc:686] T 00000000000000000000000000000000 P 55e404fbcffa45f392dc529f09a895d8 [term 1 LEADER]: Becoming Leader. State: Replica: 55e404fbcffa45f392dc529f09a895d8, State: Running, Role: LEADER
I20260929 20:23:55.994715 28050 consensus_queue.cc:237] T 00000000000000000000000000000000 P 55e404fbcffa45f392dc529f09a895d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55e404fbcffa45f392dc529f09a895d8" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 37541 } }
I20260929 20:23:55.994904 28045 sys_catalog.cc:564] T 00000000000000000000000000000000 P 55e404fbcffa45f392dc529f09a895d8 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:23:55.997604 28051 sys_catalog.cc:455] T 00000000000000000000000000000000 P 55e404fbcffa45f392dc529f09a895d8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "55e404fbcffa45f392dc529f09a895d8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55e404fbcffa45f392dc529f09a895d8" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 37541 } } }
I20260929 20:23:55.997977 28051 sys_catalog.cc:458] T 00000000000000000000000000000000 P 55e404fbcffa45f392dc529f09a895d8 [sys.catalog]: This master's current role is: LEADER
I20260929 20:23:55.998621 28052 sys_catalog.cc:455] T 00000000000000000000000000000000 P 55e404fbcffa45f392dc529f09a895d8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 55e404fbcffa45f392dc529f09a895d8. Latest consensus state: current_term: 1 leader_uuid: "55e404fbcffa45f392dc529f09a895d8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "55e404fbcffa45f392dc529f09a895d8" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 37541 } } }
I20260929 20:23:55.999161 28052 sys_catalog.cc:458] T 00000000000000000000000000000000 P 55e404fbcffa45f392dc529f09a895d8 [sys.catalog]: This master's current role is: LEADER
I20260929 20:23:55.999225 28059 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:23:56.004726 28059 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:23:56.016621 28059 catalog_manager.cc:1347] Generated new cluster ID: 7a3290c938034ca1bc45b3d934cc453c
I20260929 20:23:56.016752 28059 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:23:56.045150 28059 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 20:23:56.046535 28059 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:23:56.056731 28059 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 55e404fbcffa45f392dc529f09a895d8: Generated new TSK 0
I20260929 20:23:56.057538 28059 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.17.53.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.53.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.53.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.65.keytab.
Entry for principal kudu/127.17.53.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.65.keytab.
I20260929 20:23:56.122658 17621 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
/tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790713339161492-17621-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790713339161492-17621-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790713339161492-17621-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790713339161492-17621-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.53.65:0
--local_ip_for_outbound_sockets=127.17.53.65
--webserver_interface=127.17.53.65
--webserver_port=0
--tserver_master_addrs=127.17.53.126:37541
--keytab_file=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.65.keytab
--principal=kudu/127.17.53.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.53.84:41219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790713339161492-17621-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790713339161492-17621-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790713339161492-17621-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:23:56.500196 28071 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:23:56.500537 28071 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:23:56.500643 28071 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:23:56.508709 28071 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:23:56.508833 28071 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:23:56.508957 28071 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.53.65
I20260929 20:23:56.519367 28071 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.53.84:41219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790713339161492-17621-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790713339161492-17621-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.65.keytab
--principal=kudu/127.17.53.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.53.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790713339161492-17621-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.17.53.65
--webserver_port=0
--tserver_master_addrs=127.17.53.126:37541
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.53.65
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790713339161492-17621-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:23:56.520737 28071 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:23:56.522008 28071 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:23:56.535339 28077 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:23:56.535377 28076 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:23:56.535290 28079 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:23:56.536144 28071 server_base.cc:1034] running on GCE node
I20260929 20:23:56.536728 28071 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:23:56.537595 28071 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:23:56.538796 28071 hybrid_clock.cc:648] HybridClock initialized: now 1790713436538745 us; error 38 us; skew 500 ppm
I20260929 20:23:56.539153 28071 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:23:56 dist-test-slave-zbnc krb5kdc[27965](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713436, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.53.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:23:56.542327 28071 init.cc:377] Logged in from keytab as kudu/127.17.53.65@KRBTEST.COM (short username kudu)
I20260929 20:23:56.543970 28071 webserver.cc:460] Webserver started at http://127.17.53.65:35641/ using document root <none> and password file <none>
I20260929 20:23:56.544664 28071 fs_manager.cc:359] Metadata directory not provided
I20260929 20:23:56.544804 28071 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:23:56.545122 28071 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:23:56.548017 28071 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790713339161492-17621-0/minicluster-data/ts-0/data/instance:
uuid: "686996f5250748d889ae9fc4d7f6d756"
format_stamp: "Formatted at 2026-09-29 20:23:56 on dist-test-slave-zbnc"
server_key: "51161f8ad3818eab8bc6e7316c9c8a2a"
server_key_iv: "6b9cfcfbc3815ded4d22109b9ce49349"
server_key_version: "encryptionkey@0"
I20260929 20:23:56.548763 28071 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790713339161492-17621-0/minicluster-data/ts-0/wal/instance:
uuid: "686996f5250748d889ae9fc4d7f6d756"
format_stamp: "Formatted at 2026-09-29 20:23:56 on dist-test-slave-zbnc"
server_key: "51161f8ad3818eab8bc6e7316c9c8a2a"
server_key_iv: "6b9cfcfbc3815ded4d22109b9ce49349"
server_key_version: "encryptionkey@0"
I20260929 20:23:56.553814 28071 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260929 20:23:56.557207 28086 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:23:56.558455 28071 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260929 20:23:56.558636 28071 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790713339161492-17621-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790713339161492-17621-0/minicluster-data/ts-0/wal
uuid: "686996f5250748d889ae9fc4d7f6d756"
format_stamp: "Formatted at 2026-09-29 20:23:56 on dist-test-slave-zbnc"
server_key: "51161f8ad3818eab8bc6e7316c9c8a2a"
server_key_iv: "6b9cfcfbc3815ded4d22109b9ce49349"
server_key_version: "encryptionkey@0"
I20260929 20:23:56.558825 28071 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790713339161492-17621-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790713339161492-17621-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790713339161492-17621-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:23:56.574172 28071 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:23:56.578035 28071 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:23:56.578332 28071 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:23:56.579281 28071 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:23:56.580617 28071 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:23:56.580721 28071 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:23:56.580878 28071 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:23:56.580955 28071 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:23:56.614675 28071 rpc_server.cc:307] RPC server started. Bound to: 127.17.53.65:37263
I20260929 20:23:56.614713 28198 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.53.65:37263 every 8 connection(s)
I20260929 20:23:56.615998 28071 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790713339161492-17621-0/minicluster-data/ts-0/data/info.pb
I20260929 20:23:56.622651 17621 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu as pid 28071
I20260929 20:23:56.622931 17621 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790713339161492-17621-0/minicluster-data/ts-0/wal/instance
I20260929 20:23:56.623732 17621 external_mini_cluster.cc:1428] Setting key 7b3c35a0f9aba481a1eccd1b46b6a000
Sep 29 20:23:56 dist-test-slave-zbnc krb5kdc[27965](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713436, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.53.65@KRBTEST.COM for kudu/127.17.53.126@KRBTEST.COM
I20260929 20:23:56.638200 28201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:56.618731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.53.65:60567 (local address 127.17.53.126:37541)
0929 20:23:56.619196 (+ 465us) server_negotiation.cc:206] Beginning negotiation
0929 20:23:56.619201 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:23:56.620194 (+ 993us) server_negotiation.cc:407] Connection header received
0929 20:23:56.622018 (+ 1824us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:23:56.622022 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:23:56.622085 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:23:56.622179 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:23:56.624423 (+ 2244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:56.625338 (+ 915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:56.627771 (+ 2433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:56.628230 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:56.628333 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:56.632430 (+ 4097us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:23:56.632436 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:23:56.632439 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:23:56.632471 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:23:56.635162 (+ 2691us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:23:56.635888 (+ 726us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:23:56.635898 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:23:56.635900 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:23:56.636017 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:23:56.636453 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:23:56.636458 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:23:56.636461 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:23:56.636720 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:23:56.636874 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0929 20:23:56.637623 (+ 749us) server_negotiation.cc:299] Negotiation successful
0929 20:23:56.637841 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":361,"thread_start_us":238,"threads_started":1}
I20260929 20:23:56.639318 28202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:56.619355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:37541 (local address 127.17.53.65:60567)
0929 20:23:56.620028 (+ 673us) negotiation.cc:107] Waiting for socket to connect
0929 20:23:56.620068 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0929 20:23:56.621770 (+ 1702us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:23:56.622590 (+ 820us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:23:56.622605 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:23:56.623375 (+ 770us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:23:56.624206 (+ 831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:23:56.624243 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:23:56.625574 (+ 1331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:23:56.625578 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:23:56.627606 (+ 2028us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:23:56.627624 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:23:56.628475 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:23:56.628484 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:23:56.628816 (+ 332us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:56.629546 (+ 730us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:23:56.629579 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:23:56.632196 (+ 2617us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:23:56.635296 (+ 3100us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:23:56.635306 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:23:56.635324 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:23:56.635758 (+ 434us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:23:56.636135 (+ 377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:23:56.636140 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:23:56.636144 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:23:56.636331 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:23:56.636848 (+ 517us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:23:56.636856 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:23:56.637226 (+ 370us) client_negotiation.cc:769] Sending connection context
0929 20:23:56.637493 (+ 267us) client_negotiation.cc:240] Negotiation successful
0929 20:23:56.638052 (+ 559us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":446,"thread_start_us":254,"threads_started":1}
I20260929 20:23:56.640906 28199 heartbeater.cc:347] Connected to a master server at 127.17.53.126:37541
I20260929 20:23:56.641234 28199 heartbeater.cc:464] Registering TS with master...
I20260929 20:23:56.642170 28199 heartbeater.cc:511] Master 127.17.53.126:37541 requested a full tablet report, sending...
I20260929 20:23:56.644438 28010 ts_manager.cc:194] Registered new tserver with Master: 686996f5250748d889ae9fc4d7f6d756 (127.17.53.65:37263)
I20260929 20:23:56.645885 28010 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.53.65@KRBTEST.COM'} at 127.17.53.65:60567
WARNING: no policy specified for kudu/127.17.53.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.53.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.53.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.66.keytab.
Entry for principal kudu/127.17.53.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.66.keytab.
I20260929 20:23:56.678234 17621 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
/tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790713339161492-17621-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790713339161492-17621-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790713339161492-17621-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790713339161492-17621-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.53.66:0
--local_ip_for_outbound_sockets=127.17.53.66
--webserver_interface=127.17.53.66
--webserver_port=0
--tserver_master_addrs=127.17.53.126:37541
--keytab_file=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.66.keytab
--principal=kudu/127.17.53.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.53.84:41219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790713339161492-17621-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790713339161492-17621-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790713339161492-17621-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:23:57.005923 28205 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:23:57.006215 28205 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:23:57.006309 28205 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:23:57.013590 28205 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:23:57.013700 28205 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:23:57.013811 28205 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.53.66
I20260929 20:23:57.023631 28205 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.53.84:41219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790713339161492-17621-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790713339161492-17621-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.66.keytab
--principal=kudu/127.17.53.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.53.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790713339161492-17621-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.17.53.66
--webserver_port=0
--tserver_master_addrs=127.17.53.126:37541
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.53.66
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790713339161492-17621-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:23:57.024864 28205 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:23:57.026135 28205 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:23:57.040022 28213 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:23:57.040138 28210 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:23:57.040115 28211 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:23:57.040838 28205 server_base.cc:1034] running on GCE node
I20260929 20:23:57.041460 28205 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:23:57.042410 28205 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:23:57.043641 28205 hybrid_clock.cc:648] HybridClock initialized: now 1790713437043584 us; error 51 us; skew 500 ppm
I20260929 20:23:57.043965 28205 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:23:57 dist-test-slave-zbnc krb5kdc[27965](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713437, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.53.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:23:57.047248 28205 init.cc:377] Logged in from keytab as kudu/127.17.53.66@KRBTEST.COM (short username kudu)
I20260929 20:23:57.048672 28205 webserver.cc:460] Webserver started at http://127.17.53.66:35599/ using document root <none> and password file <none>
I20260929 20:23:57.049371 28205 fs_manager.cc:359] Metadata directory not provided
I20260929 20:23:57.049521 28205 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:23:57.049822 28205 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:23:57.054582 28205 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790713339161492-17621-0/minicluster-data/ts-1/data/instance:
uuid: "fb64fe3e83324b8abce5a83d2860fc57"
format_stamp: "Formatted at 2026-09-29 20:23:57 on dist-test-slave-zbnc"
server_key: "d0ba47730d569633bb35c9d9144eeda8"
server_key_iv: "8d08baf4256048bd72fe890513c52bd7"
server_key_version: "encryptionkey@0"
I20260929 20:23:57.055639 28205 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790713339161492-17621-0/minicluster-data/ts-1/wal/instance:
uuid: "fb64fe3e83324b8abce5a83d2860fc57"
format_stamp: "Formatted at 2026-09-29 20:23:57 on dist-test-slave-zbnc"
server_key: "d0ba47730d569633bb35c9d9144eeda8"
server_key_iv: "8d08baf4256048bd72fe890513c52bd7"
server_key_version: "encryptionkey@0"
I20260929 20:23:57.062566 28205 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260929 20:23:57.067154 28220 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:23:57.068495 28205 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 20:23:57.068688 28205 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790713339161492-17621-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790713339161492-17621-0/minicluster-data/ts-1/wal
uuid: "fb64fe3e83324b8abce5a83d2860fc57"
format_stamp: "Formatted at 2026-09-29 20:23:57 on dist-test-slave-zbnc"
server_key: "d0ba47730d569633bb35c9d9144eeda8"
server_key_iv: "8d08baf4256048bd72fe890513c52bd7"
server_key_version: "encryptionkey@0"
I20260929 20:23:57.068877 28205 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790713339161492-17621-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790713339161492-17621-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790713339161492-17621-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:23:57.088868 28205 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:23:57.092890 28205 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:23:57.093290 28205 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:23:57.094444 28205 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:23:57.095916 28205 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:23:57.096019 28205 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:23:57.096175 28205 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:23:57.096247 28205 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:23:57.128738 28205 rpc_server.cc:307] RPC server started. Bound to: 127.17.53.66:40395
I20260929 20:23:57.128782 28333 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.53.66:40395 every 8 connection(s)
I20260929 20:23:57.130071 28205 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790713339161492-17621-0/minicluster-data/ts-1/data/info.pb
I20260929 20:23:57.130530 17621 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu as pid 28205
I20260929 20:23:57.130734 17621 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790713339161492-17621-0/minicluster-data/ts-1/wal/instance
I20260929 20:23:57.131208 17621 external_mini_cluster.cc:1428] Setting key fa906d59277cbc19911fe3f33e64c782
Sep 29 20:23:57 dist-test-slave-zbnc krb5kdc[27965](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713437, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.53.66@KRBTEST.COM for kudu/127.17.53.126@KRBTEST.COM
I20260929 20:23:57.150770 28201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:57.133363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.53.66:36725 (local address 127.17.53.126:37541)
0929 20:23:57.133554 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0929 20:23:57.133558 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:23:57.135037 (+ 1479us) server_negotiation.cc:407] Connection header received
0929 20:23:57.136845 (+ 1808us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:23:57.136851 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:23:57.136933 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:23:57.137048 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:23:57.138568 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:57.139347 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:57.140928 (+ 1581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:57.141365 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:57.141494 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:57.144957 (+ 3463us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:23:57.144961 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:23:57.144964 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:23:57.144996 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:23:57.147692 (+ 2696us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:23:57.148445 (+ 753us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:23:57.148453 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:23:57.148457 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:23:57.148585 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:23:57.149027 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:23:57.149033 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:23:57.149040 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:23:57.149303 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:23:57.149459 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0929 20:23:57.150160 (+ 701us) server_negotiation.cc:299] Negotiation successful
0929 20:23:57.150391 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 20:23:57.151801 28336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:57.134057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:37541 (local address 127.17.53.66:36725)
0929 20:23:57.134836 (+ 779us) negotiation.cc:107] Waiting for socket to connect
0929 20:23:57.134925 (+ 89us) client_negotiation.cc:174] Beginning negotiation
0929 20:23:57.136570 (+ 1645us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:23:57.137302 (+ 732us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:23:57.137311 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:23:57.137824 (+ 513us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:23:57.138402 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:23:57.138418 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:23:57.139496 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:23:57.139502 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:23:57.140780 (+ 1278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:23:57.140791 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:23:57.141588 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:23:57.141595 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:23:57.141827 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:57.142520 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:23:57.142545 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:23:57.144741 (+ 2196us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:23:57.147835 (+ 3094us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:23:57.147845 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:23:57.147865 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:23:57.148320 (+ 455us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:23:57.148717 (+ 397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:23:57.148722 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:23:57.148725 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:23:57.148911 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:23:57.149432 (+ 521us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:23:57.149440 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:23:57.149791 (+ 351us) client_negotiation.cc:769] Sending connection context
0929 20:23:57.150070 (+ 279us) client_negotiation.cc:240] Negotiation successful
0929 20:23:57.150635 (+ 565us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":515,"thread_start_us":306,"threads_started":1}
WARNING: no policy specified for kudu/127.17.53.67@KRBTEST.COM; defaulting to no policy
I20260929 20:23:57.153379 28334 heartbeater.cc:347] Connected to a master server at 127.17.53.126:37541
I20260929 20:23:57.153704 28334 heartbeater.cc:464] Registering TS with master...
I20260929 20:23:57.154500 28334 heartbeater.cc:511] Master 127.17.53.126:37541 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.53.67@KRBTEST.COM" created.
I20260929 20:23:57.156031 28010 ts_manager.cc:194] Registered new tserver with Master: fb64fe3e83324b8abce5a83d2860fc57 (127.17.53.66:40395)
I20260929 20:23:57.156847 28010 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.53.66@KRBTEST.COM'} at 127.17.53.66:36725
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.53.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.67.keytab.
Entry for principal kudu/127.17.53.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.67.keytab.
I20260929 20:23:57.181998 17621 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
/tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790713339161492-17621-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790713339161492-17621-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790713339161492-17621-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790713339161492-17621-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.53.67:0
--local_ip_for_outbound_sockets=127.17.53.67
--webserver_interface=127.17.53.67
--webserver_port=0
--tserver_master_addrs=127.17.53.126:37541
--keytab_file=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.67.keytab
--principal=kudu/127.17.53.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.53.84:41219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790713339161492-17621-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790713339161492-17621-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790713339161492-17621-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:23:57.513877 28339 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:23:57.514159 28339 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:23:57.514254 28339 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:23:57.521869 28339 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:23:57.521989 28339 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:23:57.522104 28339 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.53.67
I20260929 20:23:57.532146 28339 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.53.84:41219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790713339161492-17621-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790713339161492-17621-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.67.keytab
--principal=kudu/127.17.53.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.53.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790713339161492-17621-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.17.53.67
--webserver_port=0
--tserver_master_addrs=127.17.53.126:37541
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.53.67
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790713339161492-17621-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:23:57.533488 28339 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:23:57.534770 28339 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:23:57.548579 28344 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:23:57.548552 28345 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:23:57.548662 28347 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:23:57.548983 28339 server_base.cc:1034] running on GCE node
I20260929 20:23:57.549808 28339 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:23:57.550745 28339 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:23:57.551944 28339 hybrid_clock.cc:648] HybridClock initialized: now 1790713437551892 us; error 40 us; skew 500 ppm
I20260929 20:23:57.552265 28339 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:23:57 dist-test-slave-zbnc krb5kdc[27965](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713437, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.53.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:23:57.555487 28339 init.cc:377] Logged in from keytab as kudu/127.17.53.67@KRBTEST.COM (short username kudu)
I20260929 20:23:57.556866 28339 webserver.cc:460] Webserver started at http://127.17.53.67:40895/ using document root <none> and password file <none>
I20260929 20:23:57.557490 28339 fs_manager.cc:359] Metadata directory not provided
I20260929 20:23:57.557585 28339 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:23:57.557840 28339 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:23:57.560767 28339 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790713339161492-17621-0/minicluster-data/ts-2/data/instance:
uuid: "c9c20e51055f4b6aa79516b54d6adc59"
format_stamp: "Formatted at 2026-09-29 20:23:57 on dist-test-slave-zbnc"
server_key: "609eb27658d6b096c5257a4b32c82794"
server_key_iv: "81affbfee3142f9ce4ddd4367c034868"
server_key_version: "encryptionkey@0"
I20260929 20:23:57.561458 28339 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790713339161492-17621-0/minicluster-data/ts-2/wal/instance:
uuid: "c9c20e51055f4b6aa79516b54d6adc59"
format_stamp: "Formatted at 2026-09-29 20:23:57 on dist-test-slave-zbnc"
server_key: "609eb27658d6b096c5257a4b32c82794"
server_key_iv: "81affbfee3142f9ce4ddd4367c034868"
server_key_version: "encryptionkey@0"
I20260929 20:23:57.566818 28339 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260929 20:23:57.570155 28354 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:23:57.571521 28339 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 20:23:57.571728 28339 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790713339161492-17621-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790713339161492-17621-0/minicluster-data/ts-2/wal
uuid: "c9c20e51055f4b6aa79516b54d6adc59"
format_stamp: "Formatted at 2026-09-29 20:23:57 on dist-test-slave-zbnc"
server_key: "609eb27658d6b096c5257a4b32c82794"
server_key_iv: "81affbfee3142f9ce4ddd4367c034868"
server_key_version: "encryptionkey@0"
I20260929 20:23:57.571943 28339 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790713339161492-17621-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790713339161492-17621-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790713339161492-17621-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:23:57.594189 28339 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:23:57.599038 28339 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:23:57.599314 28339 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:23:57.600224 28339 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:23:57.602375 28339 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:23:57.602451 28339 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:23:57.602535 28339 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:23:57.602578 28339 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:23:57.637334 28339 rpc_server.cc:307] RPC server started. Bound to: 127.17.53.67:40023
I20260929 20:23:57.637418 28466 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.53.67:40023 every 8 connection(s)
I20260929 20:23:57.638700 28339 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790713339161492-17621-0/minicluster-data/ts-2/data/info.pb
I20260929 20:23:57.645671 17621 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu as pid 28339
I20260929 20:23:57.645877 17621 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790713339161492-17621-0/minicluster-data/ts-2/wal/instance
I20260929 20:23:57.646380 17621 external_mini_cluster.cc:1428] Setting key 4ab4985c72fc9abcef0f506118e20dbe
I20260929 20:23:57.649200 28199 heartbeater.cc:503] Master 127.17.53.126:37541 was elected leader, sending a full tablet report...
Sep 29 20:23:57 dist-test-slave-zbnc krb5kdc[27965](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713437, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.53.67@KRBTEST.COM for kudu/127.17.53.126@KRBTEST.COM
I20260929 20:23:57.659649 28201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:57.641413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.53.67:60057 (local address 127.17.53.126:37541)
0929 20:23:57.641602 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0929 20:23:57.641607 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:23:57.642907 (+ 1300us) server_negotiation.cc:407] Connection header received
0929 20:23:57.644720 (+ 1813us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:23:57.644725 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:23:57.644790 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:23:57.644879 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:23:57.646953 (+ 2074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:57.647952 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:57.650161 (+ 2209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:57.650791 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:57.650980 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:57.654386 (+ 3406us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:23:57.654391 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:23:57.654394 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:23:57.654423 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:23:57.657024 (+ 2601us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:23:57.657697 (+ 673us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:23:57.657701 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:23:57.657703 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:23:57.657821 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:23:57.658182 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:23:57.658185 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:23:57.658187 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:23:57.658426 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:23:57.658530 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0929 20:23:57.659244 (+ 714us) server_negotiation.cc:299] Negotiation successful
0929 20:23:57.659398 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260929 20:23:57.660823 28469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:57.641968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:37541 (local address 127.17.53.67:60057)
0929 20:23:57.642720 (+ 752us) negotiation.cc:107] Waiting for socket to connect
0929 20:23:57.642759 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0929 20:23:57.644469 (+ 1710us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:23:57.645116 (+ 647us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:23:57.645124 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:23:57.645902 (+ 778us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:23:57.646721 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:23:57.646750 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:23:57.648103 (+ 1353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:23:57.648109 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:23:57.650032 (+ 1923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:23:57.650043 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:23:57.651079 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:23:57.651085 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:23:57.651274 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:57.651974 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:23:57.651995 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:23:57.654200 (+ 2205us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:23:57.657138 (+ 2938us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:23:57.657144 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:23:57.657157 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:23:57.657599 (+ 442us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:23:57.657915 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:23:57.657918 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:23:57.657920 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:23:57.658091 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:23:57.658551 (+ 460us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:23:57.658557 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:23:57.658919 (+ 362us) client_negotiation.cc:769] Sending connection context
0929 20:23:57.659161 (+ 242us) client_negotiation.cc:240] Negotiation successful
0929 20:23:57.659543 (+ 382us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":519,"thread_start_us":281,"threads_started":1}
I20260929 20:23:57.662477 28467 heartbeater.cc:347] Connected to a master server at 127.17.53.126:37541
I20260929 20:23:57.662849 28467 heartbeater.cc:464] Registering TS with master...
I20260929 20:23:57.663609 28467 heartbeater.cc:511] Master 127.17.53.126:37541 requested a full tablet report, sending...
I20260929 20:23:57.664973 28009 ts_manager.cc:194] Registered new tserver with Master: c9c20e51055f4b6aa79516b54d6adc59 (127.17.53.67:40023)
I20260929 20:23:57.665894 28009 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.53.67@KRBTEST.COM'} at 127.17.53.67:60057
I20260929 20:23:57.674480 17621 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 20:23:57.691462 28201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:57.679198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40912 (local address 127.17.53.126:37541)
0929 20:23:57.679362 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0929 20:23:57.679367 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:23:57.679760 (+ 393us) server_negotiation.cc:407] Connection header received
0929 20:23:57.679898 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:23:57.679902 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:23:57.679960 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:23:57.680042 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:23:57.681403 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:57.682144 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:57.683519 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:57.683932 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:57.684073 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:57.686374 (+ 2301us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:23:57.686379 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:23:57.686382 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:23:57.686413 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:23:57.688794 (+ 2381us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:23:57.689438 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:23:57.689445 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:23:57.689448 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:23:57.689583 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:23:57.689975 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:23:57.689981 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:23:57.689987 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:23:57.690316 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:23:57.690596 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0929 20:23:57.690938 (+ 342us) server_negotiation.cc:299] Negotiation successful
0929 20:23:57.691149 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260929 20:23:57.695751 28009 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40912:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 20:23:57.702610 28009 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 20:23:57.724838 28480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:57.714303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33184 (local address 127.17.53.67:40023)
0929 20:23:57.716578 (+ 2275us) server_negotiation.cc:206] Beginning negotiation
0929 20:23:57.716587 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 20:23:57.716629 (+ 42us) server_negotiation.cc:407] Connection header received
0929 20:23:57.716721 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:23:57.716731 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:23:57.717005 (+ 274us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:23:57.717946 (+ 941us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:23:57.718225 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:57.719927 (+ 1702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:57.722509 (+ 2582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:57.723666 (+ 1157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:57.723808 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:57.724061 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0929 20:23:57.724311 (+ 250us) server_negotiation.cc:299] Negotiation successful
0929 20:23:57.724489 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2067,"thread_start_us":1854,"threads_started":1}
I20260929 20:23:57.724874 28481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:57.714022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.67:40023 (local address 127.0.0.1:33184)
0929 20:23:57.715611 (+ 1589us) negotiation.cc:107] Waiting for socket to connect
0929 20:23:57.715658 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0929 20:23:57.715819 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:23:57.717260 (+ 1441us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:23:57.717267 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:23:57.717295 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:23:57.717759 (+ 464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:23:57.717772 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:23:57.720077 (+ 2305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:23:57.720082 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:23:57.722314 (+ 2232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:23:57.722358 (+ 44us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:23:57.723864 (+ 1506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:23:57.723871 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:23:57.724066 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:57.724121 (+ 55us) client_negotiation.cc:769] Sending connection context
0929 20:23:57.724269 (+ 148us) client_negotiation.cc:240] Negotiation successful
0929 20:23:57.724477 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1354,"mutex_wait_us":2,"thread_start_us":2813,"threads_started":1}
I20260929 20:23:57.725778 28479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:57.714487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.66:40395 (local address 127.0.0.1:37438)
0929 20:23:57.717116 (+ 2629us) negotiation.cc:107] Waiting for socket to connect
0929 20:23:57.717141 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0929 20:23:57.717251 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:23:57.718130 (+ 879us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:23:57.718134 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:23:57.718156 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:23:57.718619 (+ 463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:23:57.718638 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:23:57.720567 (+ 1929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:23:57.720571 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:23:57.722848 (+ 2277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:23:57.722899 (+ 51us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:23:57.724977 (+ 2078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:23:57.724984 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:23:57.725178 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:57.725208 (+ 30us) client_negotiation.cc:769] Sending connection context
0929 20:23:57.725295 (+ 87us) client_negotiation.cc:240] Negotiation successful
0929 20:23:57.725421 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2475,"thread_start_us":755,"threads_started":1}
I20260929 20:23:57.726508 28482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:57.715029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37438 (local address 127.17.53.66:40395)
0929 20:23:57.716866 (+ 1837us) server_negotiation.cc:206] Beginning negotiation
0929 20:23:57.716874 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 20:23:57.717362 (+ 488us) server_negotiation.cc:407] Connection header received
0929 20:23:57.717461 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:23:57.717469 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:23:57.717702 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:23:57.717889 (+ 187us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:23:57.718831 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:57.720440 (+ 1609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:57.723042 (+ 2602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:57.724509 (+ 1467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:57.724643 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:57.724841 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0929 20:23:57.726030 (+ 1189us) server_negotiation.cc:299] Negotiation successful
0929 20:23:57.726198 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1641,"thread_start_us":1499,"threads_started":1}
I20260929 20:23:57.728739 28483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:57.717412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58560 (local address 127.17.53.65:37263)
0929 20:23:57.719475 (+ 2063us) server_negotiation.cc:206] Beginning negotiation
0929 20:23:57.719482 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:23:57.719502 (+ 20us) server_negotiation.cc:407] Connection header received
0929 20:23:57.719606 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:23:57.719612 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:23:57.719776 (+ 164us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:23:57.719952 (+ 176us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:23:57.720657 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:57.722263 (+ 1606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:57.724649 (+ 2386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:57.727659 (+ 3010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:57.727813 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:57.728038 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0929 20:23:57.728257 (+ 219us) server_negotiation.cc:299] Negotiation successful
0929 20:23:57.728425 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1909,"thread_start_us":1767,"threads_started":1}
I20260929 20:23:57.728780 28484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:57.717905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.65:37263 (local address 127.0.0.1:58560)
0929 20:23:57.718656 (+ 751us) negotiation.cc:107] Waiting for socket to connect
0929 20:23:57.718688 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0929 20:23:57.718821 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:23:57.719951 (+ 1130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:23:57.719956 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:23:57.719977 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:23:57.720429 (+ 452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:23:57.720439 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:23:57.722411 (+ 1972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:23:57.722418 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:23:57.724455 (+ 2037us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:23:57.724470 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:23:57.727875 (+ 3405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:23:57.727882 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:23:57.728056 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:57.728098 (+ 42us) client_negotiation.cc:769] Sending connection context
0929 20:23:57.728203 (+ 105us) client_negotiation.cc:240] Negotiation successful
0929 20:23:57.728336 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":579,"mutex_wait_us":26,"thread_start_us":429,"threads_started":1}
I20260929 20:23:57.729462 28402 tablet_service.cc:1467] Processing CreateTablet for tablet 6caf9a8dc57f44efa5d4564bb6902d80 (DEFAULT_TABLE table=test-table [id=0362e5708f8c4158930cee61ea1f3939]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:23:57.729928 28269 tablet_service.cc:1467] Processing CreateTablet for tablet 6caf9a8dc57f44efa5d4564bb6902d80 (DEFAULT_TABLE table=test-table [id=0362e5708f8c4158930cee61ea1f3939]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:23:57.731343 28402 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6caf9a8dc57f44efa5d4564bb6902d80. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:23:57.731343 28269 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6caf9a8dc57f44efa5d4564bb6902d80. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:23:57.732334 28134 tablet_service.cc:1467] Processing CreateTablet for tablet 6caf9a8dc57f44efa5d4564bb6902d80 (DEFAULT_TABLE table=test-table [id=0362e5708f8c4158930cee61ea1f3939]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:23:57.733382 28134 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6caf9a8dc57f44efa5d4564bb6902d80. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:23:57.744222 28485 tablet_bootstrap.cc:492] T 6caf9a8dc57f44efa5d4564bb6902d80 P 686996f5250748d889ae9fc4d7f6d756: Bootstrap starting.
I20260929 20:23:57.747949 28486 tablet_bootstrap.cc:492] T 6caf9a8dc57f44efa5d4564bb6902d80 P fb64fe3e83324b8abce5a83d2860fc57: Bootstrap starting.
I20260929 20:23:57.748131 28485 tablet_bootstrap.cc:654] T 6caf9a8dc57f44efa5d4564bb6902d80 P 686996f5250748d889ae9fc4d7f6d756: Neither blocks nor log segments found. Creating new log.
I20260929 20:23:57.748767 28487 tablet_bootstrap.cc:492] T 6caf9a8dc57f44efa5d4564bb6902d80 P c9c20e51055f4b6aa79516b54d6adc59: Bootstrap starting.
I20260929 20:23:57.749636 28485 log.cc:824] T 6caf9a8dc57f44efa5d4564bb6902d80 P 686996f5250748d889ae9fc4d7f6d756: Log is configured to *not* fsync() on all Append() calls
I20260929 20:23:57.753342 28487 tablet_bootstrap.cc:654] T 6caf9a8dc57f44efa5d4564bb6902d80 P c9c20e51055f4b6aa79516b54d6adc59: Neither blocks nor log segments found. Creating new log.
I20260929 20:23:57.753896 28485 tablet_bootstrap.cc:492] T 6caf9a8dc57f44efa5d4564bb6902d80 P 686996f5250748d889ae9fc4d7f6d756: No bootstrap required, opened a new log
I20260929 20:23:57.754189 28485 ts_tablet_manager.cc:1397] T 6caf9a8dc57f44efa5d4564bb6902d80 P 686996f5250748d889ae9fc4d7f6d756: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.002s
I20260929 20:23:57.754590 28487 log.cc:824] T 6caf9a8dc57f44efa5d4564bb6902d80 P c9c20e51055f4b6aa79516b54d6adc59: Log is configured to *not* fsync() on all Append() calls
I20260929 20:23:57.754940 28486 tablet_bootstrap.cc:654] T 6caf9a8dc57f44efa5d4564bb6902d80 P fb64fe3e83324b8abce5a83d2860fc57: Neither blocks nor log segments found. Creating new log.
I20260929 20:23:57.756368 28486 log.cc:824] T 6caf9a8dc57f44efa5d4564bb6902d80 P fb64fe3e83324b8abce5a83d2860fc57: Log is configured to *not* fsync() on all Append() calls
I20260929 20:23:57.759251 28486 tablet_bootstrap.cc:492] T 6caf9a8dc57f44efa5d4564bb6902d80 P fb64fe3e83324b8abce5a83d2860fc57: No bootstrap required, opened a new log
I20260929 20:23:57.759511 28486 ts_tablet_manager.cc:1397] T 6caf9a8dc57f44efa5d4564bb6902d80 P fb64fe3e83324b8abce5a83d2860fc57: Time spent bootstrapping tablet: real 0.012s user 0.007s sys 0.000s
I20260929 20:23:57.759683 28487 tablet_bootstrap.cc:492] T 6caf9a8dc57f44efa5d4564bb6902d80 P c9c20e51055f4b6aa79516b54d6adc59: No bootstrap required, opened a new log
I20260929 20:23:57.759938 28487 ts_tablet_manager.cc:1397] T 6caf9a8dc57f44efa5d4564bb6902d80 P c9c20e51055f4b6aa79516b54d6adc59: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.001s
I20260929 20:23:57.763866 28485 raft_consensus.cc:348] T 6caf9a8dc57f44efa5d4564bb6902d80 P 686996f5250748d889ae9fc4d7f6d756 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9c20e51055f4b6aa79516b54d6adc59" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 40023 } } peers { permanent_uuid: "686996f5250748d889ae9fc4d7f6d756" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 37263 } } peers { permanent_uuid: "fb64fe3e83324b8abce5a83d2860fc57" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 40395 } }
I20260929 20:23:57.764375 28485 raft_consensus.cc:374] T 6caf9a8dc57f44efa5d4564bb6902d80 P 686996f5250748d889ae9fc4d7f6d756 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:23:57.764489 28485 raft_consensus.cc:729] T 6caf9a8dc57f44efa5d4564bb6902d80 P 686996f5250748d889ae9fc4d7f6d756 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 686996f5250748d889ae9fc4d7f6d756, State: Initialized, Role: FOLLOWER
I20260929 20:23:57.765127 28485 consensus_queue.cc:260] T 6caf9a8dc57f44efa5d4564bb6902d80 P 686996f5250748d889ae9fc4d7f6d756 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9c20e51055f4b6aa79516b54d6adc59" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 40023 } } peers { permanent_uuid: "686996f5250748d889ae9fc4d7f6d756" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 37263 } } peers { permanent_uuid: "fb64fe3e83324b8abce5a83d2860fc57" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 40395 } }
I20260929 20:23:57.768074 28485 ts_tablet_manager.cc:1428] T 6caf9a8dc57f44efa5d4564bb6902d80 P 686996f5250748d889ae9fc4d7f6d756: Time spent starting tablet: real 0.014s user 0.011s sys 0.003s
I20260929 20:23:57.767923 28486 raft_consensus.cc:348] T 6caf9a8dc57f44efa5d4564bb6902d80 P fb64fe3e83324b8abce5a83d2860fc57 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9c20e51055f4b6aa79516b54d6adc59" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 40023 } } peers { permanent_uuid: "686996f5250748d889ae9fc4d7f6d756" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 37263 } } peers { permanent_uuid: "fb64fe3e83324b8abce5a83d2860fc57" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 40395 } }
I20260929 20:23:57.768404 28486 raft_consensus.cc:374] T 6caf9a8dc57f44efa5d4564bb6902d80 P fb64fe3e83324b8abce5a83d2860fc57 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:23:57.768512 28486 raft_consensus.cc:729] T 6caf9a8dc57f44efa5d4564bb6902d80 P fb64fe3e83324b8abce5a83d2860fc57 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb64fe3e83324b8abce5a83d2860fc57, State: Initialized, Role: FOLLOWER
I20260929 20:23:57.769109 28486 consensus_queue.cc:260] T 6caf9a8dc57f44efa5d4564bb6902d80 P fb64fe3e83324b8abce5a83d2860fc57 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9c20e51055f4b6aa79516b54d6adc59" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 40023 } } peers { permanent_uuid: "686996f5250748d889ae9fc4d7f6d756" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 37263 } } peers { permanent_uuid: "fb64fe3e83324b8abce5a83d2860fc57" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 40395 } }
I20260929 20:23:57.770205 28334 heartbeater.cc:503] Master 127.17.53.126:37541 was elected leader, sending a full tablet report...
I20260929 20:23:57.770161 28487 raft_consensus.cc:348] T 6caf9a8dc57f44efa5d4564bb6902d80 P c9c20e51055f4b6aa79516b54d6adc59 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9c20e51055f4b6aa79516b54d6adc59" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 40023 } } peers { permanent_uuid: "686996f5250748d889ae9fc4d7f6d756" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 37263 } } peers { permanent_uuid: "fb64fe3e83324b8abce5a83d2860fc57" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 40395 } }
I20260929 20:23:57.770633 28487 raft_consensus.cc:374] T 6caf9a8dc57f44efa5d4564bb6902d80 P c9c20e51055f4b6aa79516b54d6adc59 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:23:57.770732 28487 raft_consensus.cc:729] T 6caf9a8dc57f44efa5d4564bb6902d80 P c9c20e51055f4b6aa79516b54d6adc59 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9c20e51055f4b6aa79516b54d6adc59, State: Initialized, Role: FOLLOWER
I20260929 20:23:57.771394 28487 consensus_queue.cc:260] T 6caf9a8dc57f44efa5d4564bb6902d80 P c9c20e51055f4b6aa79516b54d6adc59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9c20e51055f4b6aa79516b54d6adc59" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 40023 } } peers { permanent_uuid: "686996f5250748d889ae9fc4d7f6d756" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 37263 } } peers { permanent_uuid: "fb64fe3e83324b8abce5a83d2860fc57" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 40395 } }
I20260929 20:23:57.773072 28486 ts_tablet_manager.cc:1428] T 6caf9a8dc57f44efa5d4564bb6902d80 P fb64fe3e83324b8abce5a83d2860fc57: Time spent starting tablet: real 0.013s user 0.011s sys 0.000s
I20260929 20:23:57.773957 28467 heartbeater.cc:503] Master 127.17.53.126:37541 was elected leader, sending a full tablet report...
I20260929 20:23:57.774582 28487 ts_tablet_manager.cc:1428] T 6caf9a8dc57f44efa5d4564bb6902d80 P c9c20e51055f4b6aa79516b54d6adc59: Time spent starting tablet: real 0.015s user 0.011s sys 0.003s
W20260929 20:23:57.868432 28200 tablet.cc:2369] T 6caf9a8dc57f44efa5d4564bb6902d80 P 686996f5250748d889ae9fc4d7f6d756: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 20:23:57.883122 28335 tablet.cc:2369] T 6caf9a8dc57f44efa5d4564bb6902d80 P fb64fe3e83324b8abce5a83d2860fc57: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 20:23:57.890344 28468 tablet.cc:2369] T 6caf9a8dc57f44efa5d4564bb6902d80 P c9c20e51055f4b6aa79516b54d6adc59: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 20:23:57.980186 28492 raft_consensus.cc:482] T 6caf9a8dc57f44efa5d4564bb6902d80 P fb64fe3e83324b8abce5a83d2860fc57 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 20:23:57.980513 28492 raft_consensus.cc:504] T 6caf9a8dc57f44efa5d4564bb6902d80 P fb64fe3e83324b8abce5a83d2860fc57 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9c20e51055f4b6aa79516b54d6adc59" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 40023 } } peers { permanent_uuid: "686996f5250748d889ae9fc4d7f6d756" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 37263 } } peers { permanent_uuid: "fb64fe3e83324b8abce5a83d2860fc57" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 40395 } }
I20260929 20:23:57.982208 28492 leader_election.cc:288] T 6caf9a8dc57f44efa5d4564bb6902d80 P fb64fe3e83324b8abce5a83d2860fc57 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c9c20e51055f4b6aa79516b54d6adc59 (127.17.53.67:40023), 686996f5250748d889ae9fc4d7f6d756 (127.17.53.65:37263)
I20260929 20:23:57.990515 28480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:57.982793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.53.66:49975 (local address 127.17.53.67:40023)
0929 20:23:57.983074 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0929 20:23:57.983081 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:23:57.983421 (+ 340us) server_negotiation.cc:407] Connection header received
0929 20:23:57.983688 (+ 267us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:23:57.983697 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:23:57.983804 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:23:57.983919 (+ 115us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:23:57.984639 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:57.985698 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:57.988342 (+ 2644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:57.989582 (+ 1240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:57.989757 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:57.989833 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0929 20:23:57.990149 (+ 316us) server_negotiation.cc:299] Negotiation successful
0929 20:23:57.990245 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260929 20:23:57.990710 28494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:57.982659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.67:40023 (local address 127.17.53.66:49975)
0929 20:23:57.983304 (+ 645us) negotiation.cc:107] Waiting for socket to connect
0929 20:23:57.983321 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 20:23:57.983503 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:23:57.984016 (+ 513us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:23:57.984023 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:23:57.984066 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:23:57.984485 (+ 419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:23:57.984499 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:23:57.985841 (+ 1342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:23:57.985845 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:23:57.988180 (+ 2335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:23:57.988196 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:23:57.989786 (+ 1590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:23:57.989790 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:23:57.989980 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:57.990015 (+ 35us) client_negotiation.cc:769] Sending connection context
0929 20:23:57.990120 (+ 105us) client_negotiation.cc:240] Negotiation successful
0929 20:23:57.990254 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":542,"mutex_wait_us":54,"thread_start_us":381,"threads_started":1}
I20260929 20:23:57.990768 28483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:57.983702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.53.66:43337 (local address 127.17.53.65:37263)
0929 20:23:57.983915 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0929 20:23:57.983922 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:23:57.984424 (+ 502us) server_negotiation.cc:407] Connection header received
0929 20:23:57.984613 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:23:57.984618 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:23:57.984698 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:23:57.984806 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:23:57.985279 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:57.986144 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:57.988549 (+ 2405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:57.989794 (+ 1245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:57.989936 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:57.989985 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0929 20:23:57.990367 (+ 382us) server_negotiation.cc:299] Negotiation successful
0929 20:23:57.990533 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260929 20:23:57.990767 28495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:57.983485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.65:37263 (local address 127.17.53.66:43337)
0929 20:23:57.984285 (+ 800us) negotiation.cc:107] Waiting for socket to connect
0929 20:23:57.984307 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 20:23:57.984444 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:23:57.984822 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:23:57.984825 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:23:57.984844 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:23:57.985129 (+ 285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:23:57.985136 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:23:57.986279 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:23:57.986284 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:23:57.988375 (+ 2091us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:23:57.988384 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:23:57.989985 (+ 1601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:23:57.989990 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:23:57.990181 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:57.990237 (+ 56us) client_negotiation.cc:769] Sending connection context
0929 20:23:57.990314 (+ 77us) client_negotiation.cc:240] Negotiation successful
0929 20:23:57.990452 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":674,"thread_start_us":453,"threads_started":1}
I20260929 20:23:57.992023 28154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6caf9a8dc57f44efa5d4564bb6902d80" candidate_uuid: "fb64fe3e83324b8abce5a83d2860fc57" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "686996f5250748d889ae9fc4d7f6d756" is_pre_election: true
I20260929 20:23:57.992022 28422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6caf9a8dc57f44efa5d4564bb6902d80" candidate_uuid: "fb64fe3e83324b8abce5a83d2860fc57" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9c20e51055f4b6aa79516b54d6adc59" is_pre_election: true
I20260929 20:23:57.992517 28422 raft_consensus.cc:2454] T 6caf9a8dc57f44efa5d4564bb6902d80 P c9c20e51055f4b6aa79516b54d6adc59 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fb64fe3e83324b8abce5a83d2860fc57 in term 0.
I20260929 20:23:57.992532 28154 raft_consensus.cc:2454] T 6caf9a8dc57f44efa5d4564bb6902d80 P 686996f5250748d889ae9fc4d7f6d756 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fb64fe3e83324b8abce5a83d2860fc57 in term 0.
I20260929 20:23:57.993208 28225 leader_election.cc:302] T 6caf9a8dc57f44efa5d4564bb6902d80 P fb64fe3e83324b8abce5a83d2860fc57 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 686996f5250748d889ae9fc4d7f6d756, fb64fe3e83324b8abce5a83d2860fc57; no voters:
I20260929 20:23:57.993613 28492 raft_consensus.cc:2781] T 6caf9a8dc57f44efa5d4564bb6902d80 P fb64fe3e83324b8abce5a83d2860fc57 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 20:23:57.993765 28492 raft_consensus.cc:482] T 6caf9a8dc57f44efa5d4564bb6902d80 P fb64fe3e83324b8abce5a83d2860fc57 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 20:23:57.993870 28492 raft_consensus.cc:3037] T 6caf9a8dc57f44efa5d4564bb6902d80 P fb64fe3e83324b8abce5a83d2860fc57 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:23:57.996171 28492 raft_consensus.cc:504] T 6caf9a8dc57f44efa5d4564bb6902d80 P fb64fe3e83324b8abce5a83d2860fc57 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9c20e51055f4b6aa79516b54d6adc59" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 40023 } } peers { permanent_uuid: "686996f5250748d889ae9fc4d7f6d756" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 37263 } } peers { permanent_uuid: "fb64fe3e83324b8abce5a83d2860fc57" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 40395 } }
I20260929 20:23:57.996838 28492 leader_election.cc:288] T 6caf9a8dc57f44efa5d4564bb6902d80 P fb64fe3e83324b8abce5a83d2860fc57 [CANDIDATE]: Term 1 election: Requested vote from peers c9c20e51055f4b6aa79516b54d6adc59 (127.17.53.67:40023), 686996f5250748d889ae9fc4d7f6d756 (127.17.53.65:37263)
I20260929 20:23:57.997347 28422 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6caf9a8dc57f44efa5d4564bb6902d80" candidate_uuid: "fb64fe3e83324b8abce5a83d2860fc57" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c9c20e51055f4b6aa79516b54d6adc59"
I20260929 20:23:57.997392 28154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6caf9a8dc57f44efa5d4564bb6902d80" candidate_uuid: "fb64fe3e83324b8abce5a83d2860fc57" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "686996f5250748d889ae9fc4d7f6d756"
I20260929 20:23:57.997596 28422 raft_consensus.cc:3037] T 6caf9a8dc57f44efa5d4564bb6902d80 P c9c20e51055f4b6aa79516b54d6adc59 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:23:57.997617 28154 raft_consensus.cc:3037] T 6caf9a8dc57f44efa5d4564bb6902d80 P 686996f5250748d889ae9fc4d7f6d756 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:23:57.999924 28422 raft_consensus.cc:2454] T 6caf9a8dc57f44efa5d4564bb6902d80 P c9c20e51055f4b6aa79516b54d6adc59 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fb64fe3e83324b8abce5a83d2860fc57 in term 1.
I20260929 20:23:57.999948 28154 raft_consensus.cc:2454] T 6caf9a8dc57f44efa5d4564bb6902d80 P 686996f5250748d889ae9fc4d7f6d756 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fb64fe3e83324b8abce5a83d2860fc57 in term 1.
I20260929 20:23:58.000368 28225 leader_election.cc:302] T 6caf9a8dc57f44efa5d4564bb6902d80 P fb64fe3e83324b8abce5a83d2860fc57 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c9c20e51055f4b6aa79516b54d6adc59, fb64fe3e83324b8abce5a83d2860fc57; no voters:
I20260929 20:23:58.000646 28492 raft_consensus.cc:2781] T 6caf9a8dc57f44efa5d4564bb6902d80 P fb64fe3e83324b8abce5a83d2860fc57 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:23:58.001101 28492 raft_consensus.cc:686] T 6caf9a8dc57f44efa5d4564bb6902d80 P fb64fe3e83324b8abce5a83d2860fc57 [term 1 LEADER]: Becoming Leader. State: Replica: fb64fe3e83324b8abce5a83d2860fc57, State: Running, Role: LEADER
I20260929 20:23:58.001843 28492 consensus_queue.cc:237] T 6caf9a8dc57f44efa5d4564bb6902d80 P fb64fe3e83324b8abce5a83d2860fc57 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9c20e51055f4b6aa79516b54d6adc59" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 40023 } } peers { permanent_uuid: "686996f5250748d889ae9fc4d7f6d756" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 37263 } } peers { permanent_uuid: "fb64fe3e83324b8abce5a83d2860fc57" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 40395 } }
I20260929 20:23:58.006971 28008 catalog_manager.cc:5495] T 6caf9a8dc57f44efa5d4564bb6902d80 P fb64fe3e83324b8abce5a83d2860fc57 reported cstate change: term changed from 0 to 1, leader changed from <none> to fb64fe3e83324b8abce5a83d2860fc57 (127.17.53.66). New cstate: current_term: 1 leader_uuid: "fb64fe3e83324b8abce5a83d2860fc57" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c9c20e51055f4b6aa79516b54d6adc59" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 40023 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "686996f5250748d889ae9fc4d7f6d756" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 37263 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fb64fe3e83324b8abce5a83d2860fc57" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 40395 } health_report { overall_health: HEALTHY } } }
I20260929 20:23:58.029290 28482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:58.023649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37446 (local address 127.17.53.66:40395)
0929 20:23:58.023903 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0929 20:23:58.023908 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:23:58.023927 (+ 19us) server_negotiation.cc:407] Connection header received
0929 20:23:58.024155 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:23:58.024158 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:23:58.024245 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:23:58.024373 (+ 128us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 20:23:58.024717 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:58.025516 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:58.026843 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:58.027363 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:58.027469 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:58.027985 (+ 516us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 20:23:58.028655 (+ 670us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 20:23:58.028774 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0929 20:23:58.028894 (+ 120us) server_negotiation.cc:299] Negotiation successful
0929 20:23:58.028960 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102}
I20260929 20:23:58.037815 28154 raft_consensus.cc:1261] T 6caf9a8dc57f44efa5d4564bb6902d80 P 686996f5250748d889ae9fc4d7f6d756 [term 1 FOLLOWER]: Refusing update from remote peer fb64fe3e83324b8abce5a83d2860fc57: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:23:58.037828 28422 raft_consensus.cc:1261] T 6caf9a8dc57f44efa5d4564bb6902d80 P c9c20e51055f4b6aa79516b54d6adc59 [term 1 FOLLOWER]: Refusing update from remote peer fb64fe3e83324b8abce5a83d2860fc57: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:23:58.038765 28496 consensus_queue.cc:1035] T 6caf9a8dc57f44efa5d4564bb6902d80 P fb64fe3e83324b8abce5a83d2860fc57 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c9c20e51055f4b6aa79516b54d6adc59" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 40023 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:23:58.039227 28492 consensus_queue.cc:1035] T 6caf9a8dc57f44efa5d4564bb6902d80 P fb64fe3e83324b8abce5a83d2860fc57 [LEADER]: Connected to new peer: Peer: permanent_uuid: "686996f5250748d889ae9fc4d7f6d756" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 37263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:23:58.052392 28501 mvcc.cc:205] Tried to move back new op lower bound from 7334762242192728064 to 7334762242059358208. Current Snapshot: MvccSnapshot[applied={T|T < 7334762242192728064}]
I20260929 20:23:58.060861 28500 mvcc.cc:205] Tried to move back new op lower bound from 7334762242192728064 to 7334762242059358208. Current Snapshot: MvccSnapshot[applied={T|T < 7334762242192728064}]
I20260929 20:23:58.069379 28008 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40912:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 20:23:58.069674 28008 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40912:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 20:23:58.073391 28008 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 55e404fbcffa45f392dc529f09a895d8: Sending DeleteTablet for 3 replicas of tablet 6caf9a8dc57f44efa5d4564bb6902d80
I20260929 20:23:58.074751 28269 tablet_service.cc:1514] Processing DeleteTablet for tablet 6caf9a8dc57f44efa5d4564bb6902d80 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:23:58 UTC) from {username='kudu', principal='kudu/127.17.53.126@KRBTEST.COM'} at 127.0.0.1:37438
I20260929 20:23:58.075109 28134 tablet_service.cc:1514] Processing DeleteTablet for tablet 6caf9a8dc57f44efa5d4564bb6902d80 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:23:58 UTC) from {username='kudu', principal='kudu/127.17.53.126@KRBTEST.COM'} at 127.0.0.1:58560
I20260929 20:23:58.075493 28402 tablet_service.cc:1514] Processing DeleteTablet for tablet 6caf9a8dc57f44efa5d4564bb6902d80 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:23:58 UTC) from {username='kudu', principal='kudu/127.17.53.126@KRBTEST.COM'} at 127.0.0.1:33184
I20260929 20:23:58.079841 17621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu pid 28071
I20260929 20:23:58.086341 28511 tablet_replica.cc:330] T 6caf9a8dc57f44efa5d4564bb6902d80 P c9c20e51055f4b6aa79516b54d6adc59: stopping tablet replica
I20260929 20:23:58.087661 28510 tablet_replica.cc:330] T 6caf9a8dc57f44efa5d4564bb6902d80 P 686996f5250748d889ae9fc4d7f6d756: stopping tablet replica
I20260929 20:23:58.088210 28510 raft_consensus.cc:2229] T 6caf9a8dc57f44efa5d4564bb6902d80 P 686996f5250748d889ae9fc4d7f6d756 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 20:23:58.088550 28509 tablet_replica.cc:330] T 6caf9a8dc57f44efa5d4564bb6902d80 P fb64fe3e83324b8abce5a83d2860fc57: stopping tablet replica
I20260929 20:23:58.088766 28510 raft_consensus.cc:2258] T 6caf9a8dc57f44efa5d4564bb6902d80 P 686996f5250748d889ae9fc4d7f6d756 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:23:58.088950 28509 raft_consensus.cc:2229] T 6caf9a8dc57f44efa5d4564bb6902d80 P fb64fe3e83324b8abce5a83d2860fc57 [term 1 LEADER]: Raft consensus shutting down.
I20260929 20:23:58.089635 28509 raft_consensus.cc:2258] T 6caf9a8dc57f44efa5d4564bb6902d80 P fb64fe3e83324b8abce5a83d2860fc57 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:23:58.091059 28511 raft_consensus.cc:2229] T 6caf9a8dc57f44efa5d4564bb6902d80 P c9c20e51055f4b6aa79516b54d6adc59 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 20:23:58.091542 28511 raft_consensus.cc:2258] T 6caf9a8dc57f44efa5d4564bb6902d80 P c9c20e51055f4b6aa79516b54d6adc59 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:23:58.092010 28509 ts_tablet_manager.cc:1905] T 6caf9a8dc57f44efa5d4564bb6902d80 P fb64fe3e83324b8abce5a83d2860fc57: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:23:58.093658 28511 ts_tablet_manager.cc:1905] T 6caf9a8dc57f44efa5d4564bb6902d80 P c9c20e51055f4b6aa79516b54d6adc59: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:23:58.093935 28510 ts_tablet_manager.cc:1905] T 6caf9a8dc57f44efa5d4564bb6902d80 P 686996f5250748d889ae9fc4d7f6d756: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:23:58.099537 28509 ts_tablet_manager.cc:1918] T 6caf9a8dc57f44efa5d4564bb6902d80 P fb64fe3e83324b8abce5a83d2860fc57: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 20:23:58.099730 28509 log.cc:1196] T 6caf9a8dc57f44efa5d4564bb6902d80 P fb64fe3e83324b8abce5a83d2860fc57: Deleting WAL directory at /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790713339161492-17621-0/minicluster-data/ts-1/wal/wals/6caf9a8dc57f44efa5d4564bb6902d80
I20260929 20:23:58.100270 28509 ts_tablet_manager.cc:1939] T 6caf9a8dc57f44efa5d4564bb6902d80 P fb64fe3e83324b8abce5a83d2860fc57: Deleting consensus metadata
I20260929 20:23:58.100847 28511 ts_tablet_manager.cc:1918] T 6caf9a8dc57f44efa5d4564bb6902d80 P c9c20e51055f4b6aa79516b54d6adc59: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 20:23:58.101015 28511 log.cc:1196] T 6caf9a8dc57f44efa5d4564bb6902d80 P c9c20e51055f4b6aa79516b54d6adc59: Deleting WAL directory at /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790713339161492-17621-0/minicluster-data/ts-2/wal/wals/6caf9a8dc57f44efa5d4564bb6902d80
I20260929 20:23:58.101506 28511 ts_tablet_manager.cc:1939] T 6caf9a8dc57f44efa5d4564bb6902d80 P c9c20e51055f4b6aa79516b54d6adc59: Deleting consensus metadata
I20260929 20:23:58.102254 27996 catalog_manager.cc:4841] TS fb64fe3e83324b8abce5a83d2860fc57 (127.17.53.66:40395): tablet 6caf9a8dc57f44efa5d4564bb6902d80 (table test-table [id=0362e5708f8c4158930cee61ea1f3939]) successfully deleted
I20260929 20:23:58.103284 27997 catalog_manager.cc:4841] TS c9c20e51055f4b6aa79516b54d6adc59 (127.17.53.67:40023): tablet 6caf9a8dc57f44efa5d4564bb6902d80 (table test-table [id=0362e5708f8c4158930cee61ea1f3939]) successfully deleted
Sep 29 20:23:58 dist-test-slave-zbnc krb5kdc[27965](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713435, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.53.65@KRBTEST.COM
I20260929 20:23:58.106109 28510 ts_tablet_manager.cc:1918] T 6caf9a8dc57f44efa5d4564bb6902d80 P 686996f5250748d889ae9fc4d7f6d756: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 20:23:58.106269 28510 log.cc:1196] T 6caf9a8dc57f44efa5d4564bb6902d80 P 686996f5250748d889ae9fc4d7f6d756: Deleting WAL directory at /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790713339161492-17621-0/minicluster-data/ts-0/wal/wals/6caf9a8dc57f44efa5d4564bb6902d80
I20260929 20:23:58.106794 28510 ts_tablet_manager.cc:1939] T 6caf9a8dc57f44efa5d4564bb6902d80 P 686996f5250748d889ae9fc4d7f6d756: Deleting consensus metadata
I20260929 20:23:58.108358 27997 catalog_manager.cc:4841] TS 686996f5250748d889ae9fc4d7f6d756 (127.17.53.65:37263): tablet 6caf9a8dc57f44efa5d4564bb6902d80 (table test-table [id=0362e5708f8c4158930cee61ea1f3939]) successfully deleted
I20260929 20:23:58.114599 28483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:58.081851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58574 (local address 127.17.53.65:37263)
0929 20:23:58.082094 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0929 20:23:58.082103 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 20:23:58.087340 (+ 5237us) server_negotiation.cc:407] Connection header received
0929 20:23:58.087757 (+ 417us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:23:58.087762 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:23:58.087877 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:23:58.087958 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:23:58.090943 (+ 2985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:58.092083 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:58.099839 (+ 7756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:58.100544 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:58.100669 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:58.107253 (+ 6584us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:23:58.107270 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:23:58.107289 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:23:58.107340 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:23:58.111468 (+ 4128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:23:58.112251 (+ 783us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:23:58.112261 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:23:58.112265 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:23:58.112448 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:23:58.112832 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:23:58.112837 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:23:58.112840 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:23:58.113354 (+ 514us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:23:58.113691 (+ 337us) server_negotiation.cc:1035] Waiting for connection context
0929 20:23:58.113919 (+ 228us) server_negotiation.cc:299] Negotiation successful
0929 20:23:58.114157 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260929 20:23:58.115397 28194 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:23:58.247813 17621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu with pid 28071
I20260929 20:23:58.271659 17621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu pid 28205
Sep 29 20:23:58 dist-test-slave-zbnc krb5kdc[27965](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713435, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.53.66@KRBTEST.COM
I20260929 20:23:58.289289 28482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:58.272502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37452 (local address 127.17.53.66:40395)
0929 20:23:58.272715 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0929 20:23:58.272719 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:23:58.272738 (+ 19us) server_negotiation.cc:407] Connection header received
0929 20:23:58.272927 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:23:58.272931 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:23:58.272994 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:23:58.273073 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:23:58.274769 (+ 1696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:58.275859 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:58.277717 (+ 1858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:58.278261 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:58.278359 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:58.283296 (+ 4937us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:23:58.283328 (+ 32us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:23:58.283345 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:23:58.283382 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:23:58.286366 (+ 2984us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:23:58.287183 (+ 817us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:23:58.287190 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:23:58.287193 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:23:58.287334 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:23:58.287722 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:23:58.287728 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:23:58.287731 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:23:58.288084 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:23:58.288412 (+ 328us) server_negotiation.cc:1035] Waiting for connection context
0929 20:23:58.288689 (+ 277us) server_negotiation.cc:299] Negotiation successful
0929 20:23:58.288910 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260929 20:23:58.290247 28329 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:23:58.402544 17621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu with pid 28205
I20260929 20:23:58.422340 17621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu pid 28339
Sep 29 20:23:58 dist-test-slave-zbnc krb5kdc[27965](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713435, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.53.67@KRBTEST.COM
I20260929 20:23:58.437243 28480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:23:58.423126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33192 (local address 127.17.53.67:40023)
0929 20:23:58.423302 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0929 20:23:58.423309 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:23:58.423355 (+ 46us) server_negotiation.cc:407] Connection header received
0929 20:23:58.423626 (+ 271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:23:58.423631 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:23:58.423710 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:23:58.423817 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:23:58.425079 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:58.425789 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:58.427084 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:23:58.427481 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:23:58.427616 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:23:58.431954 (+ 4338us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:23:58.431969 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:23:58.431991 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:23:58.432029 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:23:58.434815 (+ 2786us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:23:58.435591 (+ 776us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:23:58.435598 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:23:58.435600 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:23:58.435754 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:23:58.436076 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:23:58.436079 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:23:58.436081 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:23:58.436405 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:23:58.436606 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0929 20:23:58.436835 (+ 229us) server_negotiation.cc:299] Negotiation successful
0929 20:23:58.436995 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 20:23:58.437908 28462 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:23:58.526093 17621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu with pid 28339
I20260929 20:23:58.542807 17621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu pid 27978
I20260929 20:23:58.543972 28040 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:23:58.651417 17621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu with pid 27978
2026-09-29T20:23:58Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (4466 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790713339161492-17621-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 20:23:58 dist-test-slave-zbnc krb5kdc[28520](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 20:23:58 dist-test-slave-zbnc krb5kdc[28520](info): set up 2 sockets
Sep 29 20:23:58 dist-test-slave-zbnc krb5kdc[28520](info): commencing operation
krb5kdc: starting...
W20260929 20:23:59.762073 17621 mini_kdc.cc:121] Time spent starting KDC: real 1.067s user 0.000s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790713339161492-17621-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790713339161492-17621-0/krb5kdc/test-user.keytab.
Sep 29 20:23:59 dist-test-slave-zbnc krb5kdc[28520](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713439, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T20:23:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T20:23:59Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.17.53.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.17.53.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.17.53.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790713339161492-17621-0/krb5kdc/rangeradmin_127.17.53.84@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.17.53.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790713339161492-17621-0/krb5kdc/rangeradmin_127.17.53.84@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.17.53.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.17.53.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.17.53.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790713339161492-17621-0/krb5kdc/rangerlookup_127.17.53.84@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.17.53.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790713339161492-17621-0/krb5kdc/rangerlookup_127.17.53.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.17.53.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.17.53.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.17.53.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790713339161492-17621-0/krb5kdc/HTTP_127.17.53.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.17.53.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790713339161492-17621-0/krb5kdc/HTTP_127.17.53.84@KRBTEST.COM.keytab.
I20260929 20:24:00.084132 17621 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790713339161492-17621-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskfEhy_t/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790713339161492-17621-0/minicluster-data/postgres -l logfile start
2026-09-29 20:24:02.640 UTC [28550] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-29 20:24:02.640 UTC [28550] LOG: listening on IPv4 address "127.17.53.84", port 45237
2026-09-29 20:24:02.644 UTC [28550] LOG: listening on Unix socket "/tmp/.s.PGSQL.45237"
2026-09-29 20:24:02.666 UTC [28553] LOG: database system was shut down at 2026-09-29 20:24:01 UTC
2026-09-29 20:24:02.670 UTC [28550] LOG: database system is ready to accept connections
I20260929 20:24:03.680801 17621 mini_postgres.cc:96] Postgres bound to 45237
2026-09-29 20:24:03.699 UTC [28561] FATAL: database "slave" does not exist
127.17.53.84:45237 - accepting connections
I20260929 20:24:03.699895 17621 mini_ranger.cc:161] Starting Ranger...
I20260929 20:24:03.726171 17621 mini_ranger.cc:84] Created miniranger Postgres user
I20260929 20:24:04.248473 17621 mini_ranger.cc:87] Created ranger Postgres database
I20260929 20:24:04.248615 17621 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790713339161492-17621-0/minicluster-data/ranger-admin
2026-09-29 20:24:04,614 [I] DB FLAVOR :POSTGRES
2026-09-29 20:24:04,615 [I] --------- Verifying Ranger DB connection ---------
2026-09-29 20:24:04,615 [I] Checking connection..
2026-09-29 20:24:04,615 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfEhy_t/build/asan/bin/postgresql.jar:/tmp/dist-test-taskfEhy_t/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.53.84:45237/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-29 20:24:04,872 [I] Checking connection passed.
2026-09-29 20:24:04,872 [I] --------- Verifying version history table ---------
2026-09-29 20:24:04,873 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfEhy_t/build/asan/bin/postgresql.jar:/tmp/dist-test-taskfEhy_t/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.53.84:45237/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 20:24:05,141 [I] Table x_db_version_h does not exist in database ranger
2026-09-29 20:24:05,142 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfEhy_t/build/asan/bin/postgresql.jar:/tmp/dist-test-taskfEhy_t/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.53.84:45237/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 20:24:05,384 [I] Table x_db_version_h does not exist in database ranger
2026-09-29 20:24:05,385 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-29 20:24:05,385 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfEhy_t/build/asan/bin/postgresql.jar:/tmp/dist-test-taskfEhy_t/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.53.84:45237/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskfEhy_t/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-29 20:24:05.651 UTC [28626] WARNING: there is no transaction in progress
2026-09-29 20:24:05,658 [I] create_dbversion_catalog.sql file imported successfully
2026-09-29 20:24:05,658 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfEhy_t/build/asan/bin/postgresql.jar:/tmp/dist-test-taskfEhy_t/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.53.84:45237/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 20:24:05,892 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-29 20:24:05,892 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-29 20:24:05,892 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfEhy_t/build/asan/bin/postgresql.jar:/tmp/dist-test-taskfEhy_t/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.53.84:45237/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-29 20:24:06,121 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfEhy_t/build/asan/bin/postgresql.jar:/tmp/dist-test-taskfEhy_t/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.53.84:45237/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-29 20:24:06,355 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfEhy_t/build/asan/bin/postgresql.jar:/tmp/dist-test-taskfEhy_t/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.53.84:45237/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-zbnc.c.gcp-upstream.internal','N') ;"
2026-09-29 20:24:06,599 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfEhy_t/build/asan/bin/postgresql.jar:/tmp/dist-test-taskfEhy_t/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.53.84:45237/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-29 20:24:06,856 [I] Table x_portal_user does not exist in database ranger
2026-09-29 20:24:06,856 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfEhy_t/build/asan/bin/postgresql.jar:/tmp/dist-test-taskfEhy_t/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.53.84:45237/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-29 20:24:07,103 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-29 20:24:07,103 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfEhy_t/build/asan/bin/postgresql.jar:/tmp/dist-test-taskfEhy_t/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.53.84:45237/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-29 20:24:07,352 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfEhy_t/build/asan/bin/postgresql.jar:/tmp/dist-test-taskfEhy_t/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.53.84:45237/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-29 20:24:07,594 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-29 20:24:07,595 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfEhy_t/build/asan/bin/postgresql.jar:/tmp/dist-test-taskfEhy_t/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.53.84:45237/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskfEhy_t/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-29 20:24:08.568 UTC [28734] WARNING: there is no transaction in progress
2026-09-29 20:24:08.585 UTC [28734] WARNING: there is no transaction in progress
2026-09-29 20:24:08.601 UTC [28734] WARNING: there is no transaction in progress
2026-09-29 20:24:08.617 UTC [28734] WARNING: there is no transaction in progress
2026-09-29 20:24:08.633 UTC [28734] WARNING: there is no transaction in progress
2026-09-29 20:24:08.652 UTC [28734] WARNING: there is no transaction in progress
2026-09-29 20:24:08.709 UTC [28734] WARNING: there is no transaction in progress
2026-09-29 20:24:08.716 UTC [28734] WARNING: there is no transaction in progress
2026-09-29 20:24:08.730 UTC [28734] WARNING: there is no transaction in progress
2026-09-29 20:24:08.741 UTC [28734] WARNING: there is no transaction in progress
2026-09-29 20:24:08.752 UTC [28734] WARNING: there is no transaction in progress
2026-09-29 20:24:08.766 UTC [28734] WARNING: there is no transaction in progress
2026-09-29 20:24:08.777 UTC [28734] WARNING: there is no transaction in progress
2026-09-29 20:24:08.783 UTC [28734] WARNING: there is no transaction in progress
2026-09-29 20:24:08.790 UTC [28734] WARNING: there is no transaction in progress
2026-09-29 20:24:08.798 UTC [28734] WARNING: there is no transaction in progress
2026-09-29 20:24:09,483 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-29 20:24:09,483 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfEhy_t/build/asan/bin/postgresql.jar:/tmp/dist-test-taskfEhy_t/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.53.84:45237/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-29 20:24:09,705 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-29 20:24:09,705 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfEhy_t/build/asan/bin/postgresql.jar:/tmp/dist-test-taskfEhy_t/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.53.84:45237/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-29 20:24:09,955 [I] Table x_portal_user already exists in database 'ranger'
2026-09-29 20:24:09,956 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfEhy_t/build/asan/bin/postgresql.jar:/tmp/dist-test-taskfEhy_t/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.53.84:45237/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-29 20:24:10,207 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-29 20:24:10,208 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfEhy_t/build/asan/bin/postgresql.jar:/tmp/dist-test-taskfEhy_t/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.53.84:45237/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-29 20:24:10,443 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfEhy_t/build/asan/bin/postgresql.jar:/tmp/dist-test-taskfEhy_t/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.53.84:45237/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-29 20:24:10,673 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfEhy_t/build/asan/bin/postgresql.jar:/tmp/dist-test-taskfEhy_t/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.53.84:45237/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-zbnc.c.gcp-upstream.internal';"
2026-09-29 20:24:10,913 [I] CORE_DB_SCHEMA import status has been updated
2026-09-29 20:24:10,914 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfEhy_t/build/asan/bin/postgresql.jar:/tmp/dist-test-taskfEhy_t/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.53.84:45237/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-29 20:24:11,151 [I] DB_PATCHES have already been applied
I20260929 20:24:11.157127 17621 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790713339161492-17621-0/minicluster-data/ranger-admin:/tmp/dist-test-taskfEhy_t/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskfEhy_t/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskfEhy_t/build/asan/bin/postgresql.jar:/tmp/dist-test-taskfEhy_t/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260929 20:24:11.157228 17621 mini_ranger.cc:193] Using host: 127.17.53.84
I20260929 20:24:11.172549 17621 mini_ranger.cc:227] Ranger admin URL: http://127.17.53.84:45387
Sep 29, 2026 8:24:11 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 29, 2026 8:24:11 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 29, 2026 8:24:11 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790713339161492-17621-0/minicluster-data/ranger-admin/ews/webapp
Sep 29, 2026 8:24:11 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790713339161492-17621-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 29, 2026 8:24:11 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790713339161492-17621-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 29, 2026 8:24:12 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790713339161492-17621-0/minicluster-data/ranger-admin/ews/webapp].
Sep 29, 2026 8:24:12 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260929 20:24:12.320111 17621 mini_ranger.cc:160] Time spent starting Ranger: real 8.620s user 0.000s sys 0.040s
Sep 29, 2026 8:24:32 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 29, 2026 8:24:32 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 29, 2026 8:24:32 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 29, 2026 8:24:32 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 29, 2026 8:24:32 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 29, 2026 8:24:32 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 29, 2026 8:24:32 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 29, 2026 8:24:32 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 29, 2026 8:24:32 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 29, 2026 8:24:32 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 29, 2026 8:24:32 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 29, 2026 8:24:32 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 29, 2026 8:24:32 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 29, 2026 8:24:32 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 29, 2026 8:24:32 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 29, 2026 8:24:32 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 29, 2026 8:24:32 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 29, 2026 8:24:33 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260929 20:24:34.464681 17621 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.17.53.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.17.53.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.17.53.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790713339161492-17621-0/krb5kdc/rangerkms_127.17.53.84@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.17.53.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790713339161492-17621-0/krb5kdc/rangerkms_127.17.53.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.17.53.84@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.17.53.84@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.17.53.84@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790713339161492-17621-0/krb5kdc/HTTP_127.17.53.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.17.53.84@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790713339161492-17621-0/krb5kdc/HTTP_127.17.53.84@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 29 20:24:34 dist-test-slave-zbnc krb5kdc[28520](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713474, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260929 20:24:34.657318 17621 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260929 20:24:34.682468 17621 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260929 20:24:35.586825 17621 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260929 20:24:35.587059 17621 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790713339161492-17621-0/minicluster-data/ranger-kms
I20260929 20:24:35.587208 17621 mini_ranger_kms.cc:227] Using postgres at 127.17.53.84:45237
2026-09-29 20:24:36,047 [I] DB FLAVOR :POSTGRES
2026-09-29 20:24:36,047 [I] --------- Verifying Ranger DB connection ---------
2026-09-29 20:24:36,048 [I] Checking connection
2026-09-29 20:24:36,048 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfEhy_t/build/asan/bin/postgresql.jar:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790713339161492-17621-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.53.84:45237/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-29 20:24:36,280 [I] connection success
2026-09-29 20:24:36,281 [I] --------- Verifying Ranger DB tables ---------
2026-09-29 20:24:36,281 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-29 20:24:36,281 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfEhy_t/build/asan/bin/postgresql.jar:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790713339161492-17621-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.53.84:45237/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-29 20:24:36,529 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-29 20:24:36,530 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-29 20:24:36,530 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-29 20:24:36,530 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfEhy_t/build/asan/bin/postgresql.jar:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790713339161492-17621-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.53.84:45237/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790713339161492-17621-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-29 20:24:36,780 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260929 20:24:37.264076 17621 mini_ranger_kms.cc:310] Created kms service
I20260929 20:24:37.443140 17621 mini_ranger_kms.cc:326] Created kudu user
I20260929 20:24:37.490703 17621 mini_ranger_kms.cc:343] Created rangerkms user
Sep 29 20:24:37 dist-test-slave-zbnc krb5kdc[28520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713474, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.17.53.84@KRBTEST.COM
I20260929 20:24:37.876370 17621 mini_ranger_kms.cc:384] Added ranger policy
I20260929 20:24:37.876677 17621 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790713339161492-17621-0/minicluster-data/ranger-kms:/tmp/dist-test-taskfEhy_t/build/asan/bin/postgresql.jar:/tmp/dist-test-taskfEhy_t/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskfEhy_t/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260929 20:24:37.876741 17621 mini_ranger_kms.cc:240] Using host: 127.17.53.84
I20260929 20:24:37.894704 17621 mini_ranger_kms.cc:276] Ranger KMS PID: 28914
I20260929 20:24:37.894994 17621 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.17.53.84:44841
Sep 29, 2026 8:24:39 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 29, 2026 8:24:39 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 29, 2026 8:24:39 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790713339161492-17621-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 29, 2026 8:24:39 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790713339161492-17621-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 29, 2026 8:24:39 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790713339161492-17621-0/minicluster-data/ranger-kms/ews/webapp].
Sep 29, 2026 8:24:39 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260929 20:24:40.308846 17621 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.652s user 0.145s sys 0.335s
I20260929 20:24:40.309096 17621 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260929 20:24:40.309325 17621 mini_ranger_kms.cc:401] 127.17.53.84:44841/kms/v1/keys
Sep 29 20:24:40 dist-test-slave-zbnc krb5kdc[28520](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790713480, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.17.53.84@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 29 20:24:40 dist-test-slave-zbnc krb5kdc[28520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713480, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.17.53.84@KRBTEST.COM for HTTP/127.17.53.84@KRBTEST.COM
Sep 29 20:24:41 dist-test-slave-zbnc krb5kdc[28520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713480, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.17.53.84@KRBTEST.COM for HTTP/127.17.53.84@KRBTEST.COM
Sep 29 20:24:43 dist-test-slave-zbnc krb5kdc[28520](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713483, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.17.53.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.53.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.53.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.126.keytab.
Entry for principal kudu/127.17.53.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.126.keytab.
I20260929 20:24:43.678608 17621 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
/tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790713339161492-17621-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790713339161492-17621-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790713339161492-17621-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790713339161492-17621-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.53.126:38725
--webserver_interface=127.17.53.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.126.keytab
--principal=kudu/127.17.53.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.53.84:43185
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.53.126:38725
--ranger_config_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790713339161492-17621-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.17.53.84:44841/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790713339161492-17621-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790713339161492-17621-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790713339161492-17621-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:24:44.093096 28965 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:24:44.093429 28965 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:24:44.093580 28965 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:24:44.102154 28965 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:24:44.102293 28965 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:24:44.102399 28965 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:24:44.102653 28965 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:24:44.102851 28965 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:24:44.118835 28965 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.53.84:43185
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790713339161492-17621-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790713339161492-17621-0/minicluster-data/master-0/wal
--ranger_kms_url=127.17.53.84:44841/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.17.53.126:38725
--ranger_config_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790713339161492-17621-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.126.keytab
--principal=kudu/127.17.53.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.53.126:38725
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790713339161492-17621-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.53.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790713339161492-17621-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:24:44.120587 28965 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:24:44.122309 28965 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:24:44.140558 28970 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:24:44.140658 28971 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:24:44.141599 28965 server_base.cc:1034] running on GCE node
W20260929 20:24:44.143131 28973 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:24:44.144018 28965 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:24:44.145671 28965 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:24:44.146941 28965 hybrid_clock.cc:648] HybridClock initialized: now 1790713484146876 us; error 57 us; skew 500 ppm
I20260929 20:24:44.147307 28965 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:24:44 dist-test-slave-zbnc krb5kdc[28520](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713484, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.53.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:24:44.150477 28965 init.cc:377] Logged in from keytab as kudu/127.17.53.126@KRBTEST.COM (short username kudu)
I20260929 20:24:44.153890 28965 webserver.cc:460] Webserver started at http://127.17.53.126:37585/ using document root <none> and password file <none>
I20260929 20:24:44.154789 28965 fs_manager.cc:359] Metadata directory not provided
I20260929 20:24:44.154963 28965 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:24:44.155339 28965 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 20:24:44 dist-test-slave-zbnc krb5kdc[28520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713484, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.53.126@KRBTEST.COM for HTTP/127.17.53.84@KRBTEST.COM
I20260929 20:24:44.217294 28965 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790713339161492-17621-0/minicluster-data/master-0/data/instance:
uuid: "0df11e4cac4e4955ac0307d6fe4614c0"
format_stamp: "Formatted at 2026-09-29 20:24:44 on dist-test-slave-zbnc"
server_key: "658da4ea36db03e6ce7a1e491301c995"
server_key_iv: "86f4a0e43e2e3436603772627641eb14"
server_key_version: "kuduclusterkey@0"
I20260929 20:24:44.218340 28965 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790713339161492-17621-0/minicluster-data/master-0/wal/instance:
uuid: "0df11e4cac4e4955ac0307d6fe4614c0"
format_stamp: "Formatted at 2026-09-29 20:24:44 on dist-test-slave-zbnc"
server_key: "658da4ea36db03e6ce7a1e491301c995"
server_key_iv: "86f4a0e43e2e3436603772627641eb14"
server_key_version: "kuduclusterkey@0"
I20260929 20:24:44.225369 28965 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.009s sys 0.001s
I20260929 20:24:44.247220 28980 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:24:44.249006 28965 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.001s
I20260929 20:24:44.249210 28965 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790713339161492-17621-0/minicluster-data/master-0/data,/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790713339161492-17621-0/minicluster-data/master-0/wal
uuid: "0df11e4cac4e4955ac0307d6fe4614c0"
format_stamp: "Formatted at 2026-09-29 20:24:44 on dist-test-slave-zbnc"
server_key: "658da4ea36db03e6ce7a1e491301c995"
server_key_iv: "86f4a0e43e2e3436603772627641eb14"
server_key_version: "kuduclusterkey@0"
I20260929 20:24:44.249447 28965 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790713339161492-17621-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790713339161492-17621-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790713339161492-17621-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:24:44.289258 28965 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:24:44.293571 28965 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:24:44.293962 28965 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:24:44.325266 28965 rpc_server.cc:307] RPC server started. Bound to: 127.17.53.126:38725
I20260929 20:24:44.325234 29031 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.53.126:38725 every 8 connection(s)
I20260929 20:24:44.326989 28965 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790713339161492-17621-0/minicluster-data/master-0/data/info.pb
I20260929 20:24:44.328763 17621 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu as pid 28965
I20260929 20:24:44.328989 17621 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790713339161492-17621-0/minicluster-data/master-0/wal/instance
I20260929 20:24:44.329810 17621 external_mini_cluster.cc:1428] Setting key 4fa78ec01cf129cce4503463392be3bf
I20260929 20:24:44.332775 29032 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 29 20:24:44 dist-test-slave-zbnc krb5kdc[28520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713483, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.53.126@KRBTEST.COM
I20260929 20:24:44.349668 29032 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0df11e4cac4e4955ac0307d6fe4614c0: Bootstrap starting.
I20260929 20:24:44.353418 29032 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0df11e4cac4e4955ac0307d6fe4614c0: Neither blocks nor log segments found. Creating new log.
I20260929 20:24:44.354898 29032 log.cc:824] T 00000000000000000000000000000000 P 0df11e4cac4e4955ac0307d6fe4614c0: Log is configured to *not* fsync() on all Append() calls
I20260929 20:24:44.357661 29035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:24:44.331950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45152 (local address 127.17.53.126:38725)
0929 20:24:44.332700 (+ 750us) server_negotiation.cc:206] Beginning negotiation
0929 20:24:44.332712 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 20:24:44.332751 (+ 39us) server_negotiation.cc:407] Connection header received
0929 20:24:44.334838 (+ 2087us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:24:44.334900 (+ 62us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:24:44.335419 (+ 519us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:24:44.335889 (+ 470us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:24:44.337822 (+ 1933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:24:44.339736 (+ 1914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:24:44.341916 (+ 2180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:24:44.343221 (+ 1305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:24:44.343468 (+ 247us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:24:44.348935 (+ 5467us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:24:44.348944 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:24:44.348962 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:24:44.349001 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:24:44.352559 (+ 3558us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:24:44.353587 (+ 1028us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:24:44.353623 (+ 36us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:24:44.353634 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:24:44.353875 (+ 241us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:24:44.354382 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:24:44.354390 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:24:44.354396 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:24:44.355259 (+ 863us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:24:44.355549 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0929 20:24:44.356150 (+ 601us) server_negotiation.cc:299] Negotiation successful
0929 20:24:44.356624 (+ 474us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":548,"thread_start_us":361,"threads_started":1}
I20260929 20:24:44.362663 29032 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0df11e4cac4e4955ac0307d6fe4614c0: No bootstrap required, opened a new log
I20260929 20:24:44.371831 29032 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0df11e4cac4e4955ac0307d6fe4614c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0df11e4cac4e4955ac0307d6fe4614c0" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 38725 } }
I20260929 20:24:44.372251 29032 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0df11e4cac4e4955ac0307d6fe4614c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:24:44.372355 29032 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0df11e4cac4e4955ac0307d6fe4614c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0df11e4cac4e4955ac0307d6fe4614c0, State: Initialized, Role: FOLLOWER
I20260929 20:24:44.373024 29032 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0df11e4cac4e4955ac0307d6fe4614c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0df11e4cac4e4955ac0307d6fe4614c0" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 38725 } }
I20260929 20:24:44.373418 29032 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0df11e4cac4e4955ac0307d6fe4614c0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 20:24:44.373747 29032 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0df11e4cac4e4955ac0307d6fe4614c0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 20:24:44.374075 29032 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0df11e4cac4e4955ac0307d6fe4614c0 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:24:44.424429 29032 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0df11e4cac4e4955ac0307d6fe4614c0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0df11e4cac4e4955ac0307d6fe4614c0" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 38725 } }
I20260929 20:24:44.425163 29032 leader_election.cc:302] T 00000000000000000000000000000000 P 0df11e4cac4e4955ac0307d6fe4614c0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0df11e4cac4e4955ac0307d6fe4614c0; no voters:
I20260929 20:24:44.425882 29032 leader_election.cc:288] T 00000000000000000000000000000000 P 0df11e4cac4e4955ac0307d6fe4614c0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 20:24:44.426035 29037 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0df11e4cac4e4955ac0307d6fe4614c0 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:24:44.428279 29037 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0df11e4cac4e4955ac0307d6fe4614c0 [term 1 LEADER]: Becoming Leader. State: Replica: 0df11e4cac4e4955ac0307d6fe4614c0, State: Running, Role: LEADER
I20260929 20:24:44.429000 29037 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0df11e4cac4e4955ac0307d6fe4614c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0df11e4cac4e4955ac0307d6fe4614c0" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 38725 } }
I20260929 20:24:44.429572 29032 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0df11e4cac4e4955ac0307d6fe4614c0 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:24:44.432054 29038 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0df11e4cac4e4955ac0307d6fe4614c0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0df11e4cac4e4955ac0307d6fe4614c0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0df11e4cac4e4955ac0307d6fe4614c0" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 38725 } } }
I20260929 20:24:44.432507 29038 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0df11e4cac4e4955ac0307d6fe4614c0 [sys.catalog]: This master's current role is: LEADER
I20260929 20:24:44.433959 29039 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0df11e4cac4e4955ac0307d6fe4614c0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0df11e4cac4e4955ac0307d6fe4614c0. Latest consensus state: current_term: 1 leader_uuid: "0df11e4cac4e4955ac0307d6fe4614c0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0df11e4cac4e4955ac0307d6fe4614c0" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 38725 } } }
I20260929 20:24:44.434378 29039 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0df11e4cac4e4955ac0307d6fe4614c0 [sys.catalog]: This master's current role is: LEADER
I20260929 20:24:44.434576 29043 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:24:44.441569 29043 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:24:44.442410 29032 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790713339161492-17621-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260929 20:24:44.461412 29043 catalog_manager.cc:1347] Generated new cluster ID: ee40066cb9904f9c8dd93a238c1a8974
I20260929 20:24:44.461614 29043 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:24:44.487543 29043 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 20:24:44.489092 29043 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:24:44.498510 29043 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0df11e4cac4e4955ac0307d6fe4614c0: Generated new TSK 0
I20260929 20:24:44.499547 29043 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.17.53.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.53.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.53.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.65.keytab.
Entry for principal kudu/127.17.53.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.65.keytab.
I20260929 20:24:45.340787 17621 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
/tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790713339161492-17621-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790713339161492-17621-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790713339161492-17621-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790713339161492-17621-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.17.53.84:44841/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.53.65:0
--local_ip_for_outbound_sockets=127.17.53.65
--webserver_interface=127.17.53.65
--webserver_port=0
--tserver_master_addrs=127.17.53.126:38725
--keytab_file=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.65.keytab
--principal=kudu/127.17.53.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.53.84:43185
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790713339161492-17621-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790713339161492-17621-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790713339161492-17621-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 20:24:45 dist-test-slave-zbnc krb5kdc[28520](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790713485, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.53.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260929 20:24:45.756773 29078 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:24:45.757113 29078 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:24:45.757287 29078 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:24:45.765846 29078 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:24:45.766060 29078 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:24:45.766265 29078 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.53.65
I20260929 20:24:45.778086 29078 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.53.84:43185
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790713339161492-17621-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790713339161492-17621-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.17.53.84:44841/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.65.keytab
--principal=kudu/127.17.53.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.53.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790713339161492-17621-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.17.53.65
--webserver_port=0
--tserver_master_addrs=127.17.53.126:38725
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.53.65
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790713339161492-17621-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:24:45.779795 29078 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:24:45.781522 29078 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:24:45.797861 29089 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:24:45.798243 29091 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:24:45.799641 29088 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:24:45.800702 29078 server_base.cc:1034] running on GCE node
I20260929 20:24:45.801503 29078 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:24:45.802640 29078 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:24:45.803895 29078 hybrid_clock.cc:648] HybridClock initialized: now 1790713485803833 us; error 58 us; skew 500 ppm
I20260929 20:24:45.804240 29078 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:24:45 dist-test-slave-zbnc krb5kdc[28520](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713485, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.53.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:24:45.807885 29078 init.cc:377] Logged in from keytab as kudu/127.17.53.65@KRBTEST.COM (short username kudu)
I20260929 20:24:45.809916 29078 webserver.cc:460] Webserver started at http://127.17.53.65:38297/ using document root <none> and password file <none>
I20260929 20:24:45.810659 29078 fs_manager.cc:359] Metadata directory not provided
I20260929 20:24:45.810806 29078 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:24:45.811183 29078 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 20:24:45 dist-test-slave-zbnc krb5kdc[28520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713485, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.53.65@KRBTEST.COM for HTTP/127.17.53.84@KRBTEST.COM
I20260929 20:24:45.834527 29078 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790713339161492-17621-0/minicluster-data/ts-0/data/instance:
uuid: "2414b15baffc47b7bdd1cd1db2e0d9d2"
format_stamp: "Formatted at 2026-09-29 20:24:45 on dist-test-slave-zbnc"
server_key: "7d4064544c645a1758c846114b2940fe"
server_key_iv: "3080e38b520f62993b0f558b72079bca"
server_key_version: "kuduclusterkey@0"
I20260929 20:24:45.835770 29078 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790713339161492-17621-0/minicluster-data/ts-0/wal/instance:
uuid: "2414b15baffc47b7bdd1cd1db2e0d9d2"
format_stamp: "Formatted at 2026-09-29 20:24:45 on dist-test-slave-zbnc"
server_key: "7d4064544c645a1758c846114b2940fe"
server_key_iv: "3080e38b520f62993b0f558b72079bca"
server_key_version: "kuduclusterkey@0"
I20260929 20:24:45.849120 29078 fs_manager.cc:691] Time spent creating directory manager: real 0.013s user 0.004s sys 0.008s
I20260929 20:24:45.890825 29098 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:24:45.893157 29078 fs_manager.cc:725] Time spent opening block manager: real 0.020s user 0.006s sys 0.013s
I20260929 20:24:45.893623 29078 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790713339161492-17621-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790713339161492-17621-0/minicluster-data/ts-0/wal
uuid: "2414b15baffc47b7bdd1cd1db2e0d9d2"
format_stamp: "Formatted at 2026-09-29 20:24:45 on dist-test-slave-zbnc"
server_key: "7d4064544c645a1758c846114b2940fe"
server_key_iv: "3080e38b520f62993b0f558b72079bca"
server_key_version: "kuduclusterkey@0"
I20260929 20:24:45.894052 29078 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790713339161492-17621-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790713339161492-17621-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790713339161492-17621-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:24:45.943104 29078 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:24:45.948799 29078 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:24:45.949239 29078 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:24:45.950625 29078 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:24:45.955590 29078 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:24:45.955714 29078 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:24:45.956005 29078 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:24:45.956128 29078 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:24:46.013715 29078 rpc_server.cc:307] RPC server started. Bound to: 127.17.53.65:39737
I20260929 20:24:46.013727 29211 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.53.65:39737 every 8 connection(s)
I20260929 20:24:46.015558 29078 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790713339161492-17621-0/minicluster-data/ts-0/data/info.pb
I20260929 20:24:46.021086 17621 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu as pid 29078
I20260929 20:24:46.021317 17621 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790713339161492-17621-0/minicluster-data/ts-0/wal/instance
I20260929 20:24:46.022287 17621 external_mini_cluster.cc:1428] Setting key 576a4e7e664e703d72e26c3b61036ad4
Sep 29 20:24:46 dist-test-slave-zbnc krb5kdc[28520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713485, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.53.65@KRBTEST.COM for kudu/127.17.53.126@KRBTEST.COM
I20260929 20:24:46.050484 29214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:24:46.023272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.53.65:49877 (local address 127.17.53.126:38725)
0929 20:24:46.024364 (+ 1092us) server_negotiation.cc:206] Beginning negotiation
0929 20:24:46.024372 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 20:24:46.028090 (+ 3718us) server_negotiation.cc:407] Connection header received
0929 20:24:46.030306 (+ 2216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:24:46.030313 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:24:46.030436 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:24:46.030526 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:24:46.033133 (+ 2607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:24:46.034296 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:24:46.036874 (+ 2578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:24:46.037731 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:24:46.037836 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:24:46.042717 (+ 4881us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:24:46.042724 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:24:46.042729 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:24:46.042773 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:24:46.046482 (+ 3709us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:24:46.047542 (+ 1060us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:24:46.047548 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:24:46.047551 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:24:46.047716 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:24:46.048275 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:24:46.048280 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:24:46.048283 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:24:46.048665 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:24:46.048809 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0929 20:24:46.049843 (+ 1034us) server_negotiation.cc:299] Negotiation successful
0929 20:24:46.050072 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":974,"thread_start_us":1842,"threads_started":1}
WARNING: no policy specified for kudu/127.17.53.66@KRBTEST.COM; defaulting to no policy
I20260929 20:24:46.053489 29215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:24:46.022758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:38725 (local address 127.17.53.65:49877)
0929 20:24:46.027521 (+ 4763us) negotiation.cc:107] Waiting for socket to connect
0929 20:24:46.027573 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0929 20:24:46.029942 (+ 2369us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:24:46.031171 (+ 1229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:24:46.031185 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:24:46.031951 (+ 766us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:24:46.032877 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:24:46.032908 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:24:46.034529 (+ 1621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:24:46.034534 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:24:46.036681 (+ 2147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:24:46.036713 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:24:46.038276 (+ 1563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:24:46.038285 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:24:46.038603 (+ 318us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:24:46.039487 (+ 884us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:24:46.039520 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:24:46.042432 (+ 2912us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:24:46.046687 (+ 4255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:24:46.046697 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:24:46.046717 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:24:46.047382 (+ 665us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:24:46.047866 (+ 484us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:24:46.047870 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:24:46.047874 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:24:46.048138 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:24:46.048917 (+ 779us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:24:46.048925 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:24:46.049389 (+ 464us) client_negotiation.cc:769] Sending connection context
0929 20:24:46.051068 (+ 1679us) client_negotiation.cc:240] Negotiation successful
0929 20:24:46.051674 (+ 606us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4431,"thread_start_us":2249,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.53.66@KRBTEST.COM" created.
I20260929 20:24:46.055963 29212 heartbeater.cc:347] Connected to a master server at 127.17.53.126:38725
I20260929 20:24:46.056293 29212 heartbeater.cc:464] Registering TS with master...
I20260929 20:24:46.057189 29212 heartbeater.cc:511] Master 127.17.53.126:38725 requested a full tablet report, sending...
I20260929 20:24:46.059988 28997 ts_manager.cc:194] Registered new tserver with Master: 2414b15baffc47b7bdd1cd1db2e0d9d2 (127.17.53.65:39737)
I20260929 20:24:46.062000 28997 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.53.65@KRBTEST.COM'} at 127.17.53.65:49877
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.53.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.66.keytab.
Entry for principal kudu/127.17.53.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.66.keytab.
I20260929 20:24:46.095448 17621 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
/tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790713339161492-17621-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790713339161492-17621-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790713339161492-17621-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790713339161492-17621-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.17.53.84:44841/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.53.66:0
--local_ip_for_outbound_sockets=127.17.53.66
--webserver_interface=127.17.53.66
--webserver_port=0
--tserver_master_addrs=127.17.53.126:38725
--keytab_file=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.66.keytab
--principal=kudu/127.17.53.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.53.84:43185
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790713339161492-17621-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790713339161492-17621-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790713339161492-17621-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 20:24:46 dist-test-slave-zbnc krb5kdc[28520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713485, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.53.126@KRBTEST.COM for HTTP/127.17.53.84@KRBTEST.COM
Sep 29 20:24:46 dist-test-slave-zbnc krb5kdc[28520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713485, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.53.126@KRBTEST.COM for HTTP/127.17.53.84@KRBTEST.COM
W20260929 20:24:46.578789 29218 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:24:46.579320 29218 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:24:46.579505 29218 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:24:46.591732 29218 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:24:46.591899 29218 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:24:46.592135 29218 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.53.66
I20260929 20:24:46.608886 29218 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.53.84:43185
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790713339161492-17621-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790713339161492-17621-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.17.53.84:44841/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.66.keytab
--principal=kudu/127.17.53.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.53.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790713339161492-17621-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.17.53.66
--webserver_port=0
--tserver_master_addrs=127.17.53.126:38725
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.53.66
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790713339161492-17621-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:24:46.610849 29218 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:24:46.612862 29218 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:24:46.636160 29225 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:24:46.654613 29226 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:24:46.656201 29228 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:24:46.656587 29218 server_base.cc:1034] running on GCE node
I20260929 20:24:46.657557 29218 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:24:46.663681 29218 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:24:46.665206 29218 hybrid_clock.cc:648] HybridClock initialized: now 1790713486665028 us; error 119 us; skew 500 ppm
I20260929 20:24:46.665757 29218 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:24:46 dist-test-slave-zbnc krb5kdc[28520](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713486, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.53.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:24:46.670691 29218 init.cc:377] Logged in from keytab as kudu/127.17.53.66@KRBTEST.COM (short username kudu)
I20260929 20:24:46.726297 29218 webserver.cc:460] Webserver started at http://127.17.53.66:39651/ using document root <none> and password file <none>
I20260929 20:24:46.727633 29218 fs_manager.cc:359] Metadata directory not provided
I20260929 20:24:46.727881 29218 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:24:46.728370 29218 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 20:24:46 dist-test-slave-zbnc krb5kdc[28520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713486, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.53.66@KRBTEST.COM for HTTP/127.17.53.84@KRBTEST.COM
I20260929 20:24:46.769212 29218 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790713339161492-17621-0/minicluster-data/ts-1/data/instance:
uuid: "7724c6b6dd244c938163a023293ef5ea"
format_stamp: "Formatted at 2026-09-29 20:24:46 on dist-test-slave-zbnc"
server_key: "2b18086bfd3e193bded6c01acb8e686d"
server_key_iv: "a761c2aa272c280e559a6f57447839c6"
server_key_version: "kuduclusterkey@0"
I20260929 20:24:46.770542 29218 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790713339161492-17621-0/minicluster-data/ts-1/wal/instance:
uuid: "7724c6b6dd244c938163a023293ef5ea"
format_stamp: "Formatted at 2026-09-29 20:24:46 on dist-test-slave-zbnc"
server_key: "2b18086bfd3e193bded6c01acb8e686d"
server_key_iv: "a761c2aa272c280e559a6f57447839c6"
server_key_version: "kuduclusterkey@0"
I20260929 20:24:46.784691 29218 fs_manager.cc:691] Time spent creating directory manager: real 0.014s user 0.004s sys 0.007s
I20260929 20:24:46.844340 29235 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:24:46.847736 29218 fs_manager.cc:725] Time spent opening block manager: real 0.014s user 0.004s sys 0.003s
I20260929 20:24:46.847950 29218 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790713339161492-17621-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790713339161492-17621-0/minicluster-data/ts-1/wal
uuid: "7724c6b6dd244c938163a023293ef5ea"
format_stamp: "Formatted at 2026-09-29 20:24:46 on dist-test-slave-zbnc"
server_key: "2b18086bfd3e193bded6c01acb8e686d"
server_key_iv: "a761c2aa272c280e559a6f57447839c6"
server_key_version: "kuduclusterkey@0"
I20260929 20:24:46.848228 29218 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790713339161492-17621-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790713339161492-17621-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790713339161492-17621-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:24:46.946973 29218 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:24:46.952800 29218 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:24:46.953256 29218 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:24:46.961122 29218 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:24:46.969417 29218 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:24:46.969609 29218 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:24:46.969802 29218 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:24:46.969910 29218 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:24:47.066056 29212 heartbeater.cc:503] Master 127.17.53.126:38725 was elected leader, sending a full tablet report...
I20260929 20:24:47.079523 29218 rpc_server.cc:307] RPC server started. Bound to: 127.17.53.66:39351
I20260929 20:24:47.079654 29354 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.53.66:39351 every 8 connection(s)
I20260929 20:24:47.081337 29218 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790713339161492-17621-0/minicluster-data/ts-1/data/info.pb
I20260929 20:24:47.086277 17621 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu as pid 29218
I20260929 20:24:47.086488 17621 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790713339161492-17621-0/minicluster-data/ts-1/wal/instance
I20260929 20:24:47.087180 17621 external_mini_cluster.cc:1428] Setting key 01322241d7143311f4fcea30e1a44247
Sep 29 20:24:47 dist-test-slave-zbnc krb5kdc[28520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713486, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.53.66@KRBTEST.COM for kudu/127.17.53.126@KRBTEST.COM
I20260929 20:24:47.114688 29358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:24:47.088065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.53.66:35747 (local address 127.17.53.126:38725)
0929 20:24:47.089986 (+ 1921us) server_negotiation.cc:206] Beginning negotiation
0929 20:24:47.089992 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:24:47.090012 (+ 20us) server_negotiation.cc:407] Connection header received
0929 20:24:47.094396 (+ 4384us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:24:47.094402 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:24:47.094483 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:24:47.094581 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:24:47.096759 (+ 2178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:24:47.097876 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:24:47.100327 (+ 2451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:24:47.100921 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:24:47.101023 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:24:47.105751 (+ 4728us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:24:47.105757 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:24:47.105772 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:24:47.105812 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:24:47.109513 (+ 3701us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:24:47.110562 (+ 1049us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:24:47.110568 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:24:47.110572 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:24:47.110722 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:24:47.111292 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:24:47.111297 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:24:47.111300 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:24:47.111646 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:24:47.111819 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0929 20:24:47.114088 (+ 2269us) server_negotiation.cc:299] Negotiation successful
0929 20:24:47.114322 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1785,"thread_start_us":4824,"threads_started":1}
I20260929 20:24:47.116050 29357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:24:47.085295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:38725 (local address 127.17.53.66:35747)
0929 20:24:47.087673 (+ 2378us) negotiation.cc:107] Waiting for socket to connect
0929 20:24:47.087735 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0929 20:24:47.093954 (+ 6219us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:24:47.095031 (+ 1077us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:24:47.095043 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:24:47.095738 (+ 695us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:24:47.096555 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:24:47.096580 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:24:47.098077 (+ 1497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:24:47.098082 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:24:47.100163 (+ 2081us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:24:47.100177 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:24:47.101365 (+ 1188us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:24:47.101371 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:24:47.101648 (+ 277us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:24:47.102483 (+ 835us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:24:47.102512 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:24:47.105458 (+ 2946us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:24:47.109685 (+ 4227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:24:47.109695 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:24:47.109715 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:24:47.110312 (+ 597us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:24:47.110886 (+ 574us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:24:47.110891 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:24:47.110895 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:24:47.111142 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:24:47.111893 (+ 751us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:24:47.111901 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:24:47.112360 (+ 459us) client_negotiation.cc:769] Sending connection context
0929 20:24:47.112662 (+ 302us) client_negotiation.cc:240] Negotiation successful
0929 20:24:47.113233 (+ 571us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2049,"thread_start_us":3113,"threads_started":1}
I20260929 20:24:47.118196 29355 heartbeater.cc:347] Connected to a master server at 127.17.53.126:38725
I20260929 20:24:47.118557 29355 heartbeater.cc:464] Registering TS with master...
I20260929 20:24:47.119608 29355 heartbeater.cc:511] Master 127.17.53.126:38725 requested a full tablet report, sending...
I20260929 20:24:47.121200 28997 ts_manager.cc:194] Registered new tserver with Master: 7724c6b6dd244c938163a023293ef5ea (127.17.53.66:39351)
I20260929 20:24:47.122239 28997 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.53.66@KRBTEST.COM'} at 127.17.53.66:35747
WARNING: no policy specified for kudu/127.17.53.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.53.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.53.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.67.keytab.
Entry for principal kudu/127.17.53.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.67.keytab.
I20260929 20:24:47.163297 17621 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
/tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790713339161492-17621-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790713339161492-17621-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790713339161492-17621-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790713339161492-17621-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.17.53.84:44841/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.53.67:0
--local_ip_for_outbound_sockets=127.17.53.67
--webserver_interface=127.17.53.67
--webserver_port=0
--tserver_master_addrs=127.17.53.126:38725
--keytab_file=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.67.keytab
--principal=kudu/127.17.53.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.53.84:43185
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790713339161492-17621-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790713339161492-17621-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790713339161492-17621-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:24:47.586704 29361 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:24:47.587081 29361 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:24:47.587188 29361 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:24:47.595456 29361 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:24:47.595577 29361 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:24:47.595700 29361 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.53.67
I20260929 20:24:47.609150 29361 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.53.84:43185
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790713339161492-17621-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790713339161492-17621-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.17.53.84:44841/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.67.keytab
--principal=kudu/127.17.53.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.53.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790713339161492-17621-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.17.53.67
--webserver_port=0
--tserver_master_addrs=127.17.53.126:38725
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.53.67
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790713339161492-17621-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:24:47.610570 29361 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:24:47.612053 29361 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:24:47.627279 29366 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:24:47.627314 29369 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:24:47.627549 29367 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:24:47.628934 29361 server_base.cc:1034] running on GCE node
I20260929 20:24:47.629726 29361 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:24:47.630810 29361 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:24:47.632050 29361 hybrid_clock.cc:648] HybridClock initialized: now 1790713487631990 us; error 56 us; skew 500 ppm
I20260929 20:24:47.632359 29361 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:24:47 dist-test-slave-zbnc krb5kdc[28520](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713487, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.53.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:24:47.636022 29361 init.cc:377] Logged in from keytab as kudu/127.17.53.67@KRBTEST.COM (short username kudu)
I20260929 20:24:47.637615 29361 webserver.cc:460] Webserver started at http://127.17.53.67:37233/ using document root <none> and password file <none>
I20260929 20:24:47.638370 29361 fs_manager.cc:359] Metadata directory not provided
I20260929 20:24:47.638530 29361 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:24:47.638841 29361 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 20:24:47 dist-test-slave-zbnc krb5kdc[28520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713487, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.53.67@KRBTEST.COM for HTTP/127.17.53.84@KRBTEST.COM
I20260929 20:24:47.661540 29361 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790713339161492-17621-0/minicluster-data/ts-2/data/instance:
uuid: "398106ca592f4051a04f795b1dadc73a"
format_stamp: "Formatted at 2026-09-29 20:24:47 on dist-test-slave-zbnc"
server_key: "51d3c831f487455387ad32bbf259a36a"
server_key_iv: "fcb87fbe944a53193a412c7b97ec8276"
server_key_version: "kuduclusterkey@0"
I20260929 20:24:47.662809 29361 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790713339161492-17621-0/minicluster-data/ts-2/wal/instance:
uuid: "398106ca592f4051a04f795b1dadc73a"
format_stamp: "Formatted at 2026-09-29 20:24:47 on dist-test-slave-zbnc"
server_key: "51d3c831f487455387ad32bbf259a36a"
server_key_iv: "fcb87fbe944a53193a412c7b97ec8276"
server_key_version: "kuduclusterkey@0"
I20260929 20:24:47.671136 29361 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.007s sys 0.001s
I20260929 20:24:47.699280 29376 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:24:47.703018 29361 fs_manager.cc:725] Time spent opening block manager: real 0.013s user 0.008s sys 0.005s
I20260929 20:24:47.703274 29361 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790713339161492-17621-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790713339161492-17621-0/minicluster-data/ts-2/wal
uuid: "398106ca592f4051a04f795b1dadc73a"
format_stamp: "Formatted at 2026-09-29 20:24:47 on dist-test-slave-zbnc"
server_key: "51d3c831f487455387ad32bbf259a36a"
server_key_iv: "fcb87fbe944a53193a412c7b97ec8276"
server_key_version: "kuduclusterkey@0"
I20260929 20:24:47.703691 29361 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790713339161492-17621-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790713339161492-17621-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790713339161492-17621-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:24:47.744890 29361 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:24:47.749747 29361 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:24:47.750108 29361 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:24:47.751415 29361 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:24:47.753117 29361 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:24:47.753283 29361 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:24:47.753607 29361 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:24:47.753815 29361 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260929 20:24:47.807277 29361 rpc_server.cc:307] RPC server started. Bound to: 127.17.53.67:38875
I20260929 20:24:47.807368 29488 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.53.67:38875 every 8 connection(s)
I20260929 20:24:47.808800 29361 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790713339161492-17621-0/minicluster-data/ts-2/data/info.pb
I20260929 20:24:47.817788 17621 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu as pid 29361
I20260929 20:24:47.817961 17621 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790713339161492-17621-0/minicluster-data/ts-2/wal/instance
I20260929 20:24:47.818384 17621 external_mini_cluster.cc:1428] Setting key 7bf9e21bdead6f79ad871891d8738940
Sep 29 20:24:47 dist-test-slave-zbnc krb5kdc[28520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713487, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.53.67@KRBTEST.COM for kudu/127.17.53.126@KRBTEST.COM
I20260929 20:24:47.837872 29491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:24:47.812021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.53.67:57995 (local address 127.17.53.126:38725)
0929 20:24:47.813018 (+ 997us) server_negotiation.cc:206] Beginning negotiation
0929 20:24:47.813029 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 20:24:47.814051 (+ 1022us) server_negotiation.cc:407] Connection header received
0929 20:24:47.817573 (+ 3522us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:24:47.817581 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:24:47.817692 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:24:47.817820 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:24:47.820519 (+ 2699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:24:47.821642 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:24:47.824690 (+ 3048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:24:47.825098 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:24:47.825213 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:24:47.830261 (+ 5048us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:24:47.830265 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:24:47.830268 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:24:47.830308 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:24:47.833917 (+ 3609us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:24:47.834743 (+ 826us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:24:47.834749 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:24:47.834753 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:24:47.835012 (+ 259us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:24:47.835620 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:24:47.835623 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:24:47.835625 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:24:47.836020 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:24:47.836137 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0929 20:24:47.837333 (+ 1196us) server_negotiation.cc:299] Negotiation successful
0929 20:24:47.837534 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":785,"thread_start_us":608,"threads_started":1}
I20260929 20:24:47.840080 29492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:24:47.812845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:38725 (local address 127.17.53.67:57995)
0929 20:24:47.813840 (+ 995us) negotiation.cc:107] Waiting for socket to connect
0929 20:24:47.813909 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0929 20:24:47.817093 (+ 3184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:24:47.818127 (+ 1034us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:24:47.818144 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:24:47.819307 (+ 1163us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:24:47.820328 (+ 1021us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:24:47.820360 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:24:47.821795 (+ 1435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:24:47.821805 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:24:47.824529 (+ 2724us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:24:47.824550 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:24:47.825349 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:24:47.825367 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:24:47.825802 (+ 435us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:24:47.826619 (+ 817us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:24:47.826660 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:24:47.830008 (+ 3348us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:24:47.834063 (+ 4055us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:24:47.834072 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:24:47.834089 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:24:47.834566 (+ 477us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:24:47.835138 (+ 572us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:24:47.835146 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:24:47.835153 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:24:47.835489 (+ 336us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:24:47.836187 (+ 698us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:24:47.836199 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:24:47.836860 (+ 661us) client_negotiation.cc:769] Sending connection context
0929 20:24:47.837246 (+ 386us) client_negotiation.cc:240] Negotiation successful
0929 20:24:47.838255 (+ 1009us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":618,"thread_start_us":340,"threads_started":1}
I20260929 20:24:47.841624 29489 heartbeater.cc:347] Connected to a master server at 127.17.53.126:38725
I20260929 20:24:47.841940 29489 heartbeater.cc:464] Registering TS with master...
I20260929 20:24:47.842662 29489 heartbeater.cc:511] Master 127.17.53.126:38725 requested a full tablet report, sending...
I20260929 20:24:47.844375 28997 ts_manager.cc:194] Registered new tserver with Master: 398106ca592f4051a04f795b1dadc73a (127.17.53.67:38875)
I20260929 20:24:47.845973 28997 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.53.67@KRBTEST.COM'} at 127.17.53.67:57995
I20260929 20:24:47.847368 17621 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 20:24:47.872567 29491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:24:47.854696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45162 (local address 127.17.53.126:38725)
0929 20:24:47.854935 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0929 20:24:47.854943 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 20:24:47.855732 (+ 789us) server_negotiation.cc:407] Connection header received
0929 20:24:47.855943 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:24:47.855947 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:24:47.856030 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:24:47.856108 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:24:47.858469 (+ 2361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:24:47.859457 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:24:47.862003 (+ 2546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:24:47.862598 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:24:47.862725 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:24:47.866412 (+ 3687us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:24:47.866416 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:24:47.866419 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:24:47.866451 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:24:47.869652 (+ 3201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:24:47.870612 (+ 960us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:24:47.870615 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:24:47.870617 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:24:47.870734 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:24:47.871220 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:24:47.871223 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:24:47.871225 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:24:47.871502 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:24:47.871786 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0929 20:24:47.872150 (+ 364us) server_negotiation.cc:299] Negotiation successful
0929 20:24:47.872317 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260929 20:24:47.877770 28997 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45162:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 20:24:47.885574 28997 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 20:24:47.933555 29504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:24:47.901736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.65:39737 (local address 127.0.0.1:37944)
0929 20:24:47.920280 (+ 18544us) negotiation.cc:107] Waiting for socket to connect
0929 20:24:47.920332 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0929 20:24:47.920559 (+ 227us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:24:47.921949 (+ 1390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:24:47.921958 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:24:47.921994 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:24:47.922835 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:24:47.922884 (+ 49us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:24:47.925800 (+ 2916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:24:47.925810 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:24:47.929421 (+ 3611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:24:47.929440 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:24:47.932452 (+ 3012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:24:47.932459 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:24:47.932672 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:24:47.932730 (+ 58us) client_negotiation.cc:769] Sending connection context
0929 20:24:47.932874 (+ 144us) client_negotiation.cc:240] Negotiation successful
0929 20:24:47.933114 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18265,"thread_start_us":19366,"threads_started":1}
I20260929 20:24:47.935467 29503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:24:47.901996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37944 (local address 127.17.53.65:39737)
0929 20:24:47.918136 (+ 16140us) server_negotiation.cc:206] Beginning negotiation
0929 20:24:47.918144 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 20:24:47.920745 (+ 2601us) server_negotiation.cc:407] Connection header received
0929 20:24:47.920930 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:24:47.920942 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:24:47.921293 (+ 351us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:24:47.921558 (+ 265us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:24:47.923174 (+ 1616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:24:47.925524 (+ 2350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:24:47.929726 (+ 4202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:24:47.931850 (+ 2124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:24:47.932004 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:24:47.932286 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0929 20:24:47.934643 (+ 2357us) server_negotiation.cc:299] Negotiation successful
0929 20:24:47.934906 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15948,"thread_start_us":15750,"threads_started":1}
I20260929 20:24:47.940647 29505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:24:47.902628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52706 (local address 127.17.53.66:39351)
0929 20:24:47.919584 (+ 16956us) server_negotiation.cc:206] Beginning negotiation
0929 20:24:47.919600 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 20:24:47.927987 (+ 8387us) server_negotiation.cc:407] Connection header received
0929 20:24:47.928702 (+ 715us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:24:47.928712 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:24:47.929024 (+ 312us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:24:47.929332 (+ 308us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:24:47.931902 (+ 2570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:24:47.933482 (+ 1580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:24:47.936815 (+ 3333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:24:47.939035 (+ 2220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:24:47.939188 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:24:47.939419 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0929 20:24:47.940066 (+ 647us) server_negotiation.cc:299] Negotiation successful
0929 20:24:47.940261 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16704,"thread_start_us":16526,"threads_started":1}
I20260929 20:24:47.941659 29502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:24:47.902289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.66:39351 (local address 127.0.0.1:52706)
0929 20:24:47.927836 (+ 25547us) negotiation.cc:107] Waiting for socket to connect
0929 20:24:47.927866 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0929 20:24:47.928275 (+ 409us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:24:47.929713 (+ 1438us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:24:47.929718 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:24:47.929746 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:24:47.931355 (+ 1609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:24:47.931377 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:24:47.933730 (+ 2353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:24:47.933738 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:24:47.936629 (+ 2891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:24:47.936653 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:24:47.939612 (+ 2959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:24:47.939619 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:24:47.939830 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:24:47.939867 (+ 37us) client_negotiation.cc:769] Sending connection context
0929 20:24:47.941055 (+ 1188us) client_negotiation.cc:240] Negotiation successful
0929 20:24:47.941222 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19071,"thread_start_us":17539,"threads_started":1}
I20260929 20:24:47.946926 29290 tablet_service.cc:1467] Processing CreateTablet for tablet 99126d56ba5f4e0890042aacc51d9e24 (DEFAULT_TABLE table=test-table [id=aae83d1fae864387888308fbba55bfd0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:24:47.948711 29290 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 99126d56ba5f4e0890042aacc51d9e24. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:24:47.955274 29504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:24:47.921626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.67:38875 (local address 127.0.0.1:57674)
0929 20:24:47.934271 (+ 12645us) negotiation.cc:107] Waiting for socket to connect
0929 20:24:47.934306 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0929 20:24:47.934444 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:24:47.945554 (+ 11110us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:24:47.945561 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:24:47.945590 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:24:47.945824 (+ 234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:24:47.945838 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:24:47.948615 (+ 2777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:24:47.948621 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:24:47.950610 (+ 1989us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:24:47.950627 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:24:47.954064 (+ 3437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:24:47.954077 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:24:47.954356 (+ 279us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:24:47.954396 (+ 40us) client_negotiation.cc:769] Sending connection context
0929 20:24:47.954518 (+ 122us) client_negotiation.cc:240] Negotiation successful
0929 20:24:47.954675 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12451,"thread_start_us":30435,"threads_started":1}
I20260929 20:24:47.956641 29507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:24:47.922015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57674 (local address 127.17.53.67:38875)
0929 20:24:47.944637 (+ 22622us) server_negotiation.cc:206] Beginning negotiation
0929 20:24:47.944648 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 20:24:47.944694 (+ 46us) server_negotiation.cc:407] Connection header received
0929 20:24:47.944807 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:24:47.944816 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:24:47.945091 (+ 275us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:24:47.946029 (+ 938us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:24:47.946335 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:24:47.948409 (+ 2074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:24:47.950931 (+ 2522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:24:47.952795 (+ 1864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:24:47.952973 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:24:47.953287 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0929 20:24:47.956020 (+ 2733us) server_negotiation.cc:299] Negotiation successful
0929 20:24:47.956236 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22411,"thread_start_us":23249,"threads_started":1}
I20260929 20:24:47.957499 29146 tablet_service.cc:1467] Processing CreateTablet for tablet 99126d56ba5f4e0890042aacc51d9e24 (DEFAULT_TABLE table=test-table [id=aae83d1fae864387888308fbba55bfd0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:24:47.960596 29146 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 99126d56ba5f4e0890042aacc51d9e24. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:24:47.960731 29424 tablet_service.cc:1467] Processing CreateTablet for tablet 99126d56ba5f4e0890042aacc51d9e24 (DEFAULT_TABLE table=test-table [id=aae83d1fae864387888308fbba55bfd0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:24:47.962406 29424 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 99126d56ba5f4e0890042aacc51d9e24. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:24:47.980669 29508 tablet_bootstrap.cc:492] T 99126d56ba5f4e0890042aacc51d9e24 P 7724c6b6dd244c938163a023293ef5ea: Bootstrap starting.
I20260929 20:24:47.981026 29509 tablet_bootstrap.cc:492] T 99126d56ba5f4e0890042aacc51d9e24 P 2414b15baffc47b7bdd1cd1db2e0d9d2: Bootstrap starting.
I20260929 20:24:47.984815 29508 tablet_bootstrap.cc:654] T 99126d56ba5f4e0890042aacc51d9e24 P 7724c6b6dd244c938163a023293ef5ea: Neither blocks nor log segments found. Creating new log.
I20260929 20:24:47.984817 29509 tablet_bootstrap.cc:654] T 99126d56ba5f4e0890042aacc51d9e24 P 2414b15baffc47b7bdd1cd1db2e0d9d2: Neither blocks nor log segments found. Creating new log.
I20260929 20:24:47.986594 29509 log.cc:824] T 99126d56ba5f4e0890042aacc51d9e24 P 2414b15baffc47b7bdd1cd1db2e0d9d2: Log is configured to *not* fsync() on all Append() calls
I20260929 20:24:47.986914 29508 log.cc:824] T 99126d56ba5f4e0890042aacc51d9e24 P 7724c6b6dd244c938163a023293ef5ea: Log is configured to *not* fsync() on all Append() calls
I20260929 20:24:47.988369 29510 tablet_bootstrap.cc:492] T 99126d56ba5f4e0890042aacc51d9e24 P 398106ca592f4051a04f795b1dadc73a: Bootstrap starting.
I20260929 20:24:47.992416 29508 tablet_bootstrap.cc:492] T 99126d56ba5f4e0890042aacc51d9e24 P 7724c6b6dd244c938163a023293ef5ea: No bootstrap required, opened a new log
I20260929 20:24:47.992715 29508 ts_tablet_manager.cc:1397] T 99126d56ba5f4e0890042aacc51d9e24 P 7724c6b6dd244c938163a023293ef5ea: Time spent bootstrapping tablet: real 0.012s user 0.010s sys 0.000s
I20260929 20:24:47.992980 29510 tablet_bootstrap.cc:654] T 99126d56ba5f4e0890042aacc51d9e24 P 398106ca592f4051a04f795b1dadc73a: Neither blocks nor log segments found. Creating new log.
I20260929 20:24:47.994462 29510 log.cc:824] T 99126d56ba5f4e0890042aacc51d9e24 P 398106ca592f4051a04f795b1dadc73a: Log is configured to *not* fsync() on all Append() calls
I20260929 20:24:47.998296 29509 tablet_bootstrap.cc:492] T 99126d56ba5f4e0890042aacc51d9e24 P 2414b15baffc47b7bdd1cd1db2e0d9d2: No bootstrap required, opened a new log
I20260929 20:24:47.998651 29509 ts_tablet_manager.cc:1397] T 99126d56ba5f4e0890042aacc51d9e24 P 2414b15baffc47b7bdd1cd1db2e0d9d2: Time spent bootstrapping tablet: real 0.018s user 0.006s sys 0.006s
I20260929 20:24:48.004395 29510 tablet_bootstrap.cc:492] T 99126d56ba5f4e0890042aacc51d9e24 P 398106ca592f4051a04f795b1dadc73a: No bootstrap required, opened a new log
I20260929 20:24:48.004859 29510 ts_tablet_manager.cc:1397] T 99126d56ba5f4e0890042aacc51d9e24 P 398106ca592f4051a04f795b1dadc73a: Time spent bootstrapping tablet: real 0.017s user 0.010s sys 0.004s
I20260929 20:24:48.005740 29508 raft_consensus.cc:348] T 99126d56ba5f4e0890042aacc51d9e24 P 7724c6b6dd244c938163a023293ef5ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2414b15baffc47b7bdd1cd1db2e0d9d2" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 39737 } } peers { permanent_uuid: "398106ca592f4051a04f795b1dadc73a" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 38875 } } peers { permanent_uuid: "7724c6b6dd244c938163a023293ef5ea" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 39351 } }
I20260929 20:24:48.006330 29508 raft_consensus.cc:374] T 99126d56ba5f4e0890042aacc51d9e24 P 7724c6b6dd244c938163a023293ef5ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:24:48.006486 29508 raft_consensus.cc:729] T 99126d56ba5f4e0890042aacc51d9e24 P 7724c6b6dd244c938163a023293ef5ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7724c6b6dd244c938163a023293ef5ea, State: Initialized, Role: FOLLOWER
I20260929 20:24:48.007318 29508 consensus_queue.cc:260] T 99126d56ba5f4e0890042aacc51d9e24 P 7724c6b6dd244c938163a023293ef5ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2414b15baffc47b7bdd1cd1db2e0d9d2" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 39737 } } peers { permanent_uuid: "398106ca592f4051a04f795b1dadc73a" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 38875 } } peers { permanent_uuid: "7724c6b6dd244c938163a023293ef5ea" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 39351 } }
I20260929 20:24:48.020653 29355 heartbeater.cc:503] Master 127.17.53.126:38725 was elected leader, sending a full tablet report...
I20260929 20:24:48.021749 29508 ts_tablet_manager.cc:1428] T 99126d56ba5f4e0890042aacc51d9e24 P 7724c6b6dd244c938163a023293ef5ea: Time spent starting tablet: real 0.029s user 0.018s sys 0.008s
I20260929 20:24:48.023180 29510 raft_consensus.cc:348] T 99126d56ba5f4e0890042aacc51d9e24 P 398106ca592f4051a04f795b1dadc73a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2414b15baffc47b7bdd1cd1db2e0d9d2" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 39737 } } peers { permanent_uuid: "398106ca592f4051a04f795b1dadc73a" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 38875 } } peers { permanent_uuid: "7724c6b6dd244c938163a023293ef5ea" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 39351 } }
I20260929 20:24:48.023579 29510 raft_consensus.cc:374] T 99126d56ba5f4e0890042aacc51d9e24 P 398106ca592f4051a04f795b1dadc73a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:24:48.023710 29510 raft_consensus.cc:729] T 99126d56ba5f4e0890042aacc51d9e24 P 398106ca592f4051a04f795b1dadc73a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 398106ca592f4051a04f795b1dadc73a, State: Initialized, Role: FOLLOWER
I20260929 20:24:48.024370 29510 consensus_queue.cc:260] T 99126d56ba5f4e0890042aacc51d9e24 P 398106ca592f4051a04f795b1dadc73a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2414b15baffc47b7bdd1cd1db2e0d9d2" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 39737 } } peers { permanent_uuid: "398106ca592f4051a04f795b1dadc73a" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 38875 } } peers { permanent_uuid: "7724c6b6dd244c938163a023293ef5ea" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 39351 } }
I20260929 20:24:48.025496 29489 heartbeater.cc:503] Master 127.17.53.126:38725 was elected leader, sending a full tablet report...
I20260929 20:24:48.026268 29510 ts_tablet_manager.cc:1428] T 99126d56ba5f4e0890042aacc51d9e24 P 398106ca592f4051a04f795b1dadc73a: Time spent starting tablet: real 0.021s user 0.016s sys 0.000s
I20260929 20:24:48.035732 29509 raft_consensus.cc:348] T 99126d56ba5f4e0890042aacc51d9e24 P 2414b15baffc47b7bdd1cd1db2e0d9d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2414b15baffc47b7bdd1cd1db2e0d9d2" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 39737 } } peers { permanent_uuid: "398106ca592f4051a04f795b1dadc73a" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 38875 } } peers { permanent_uuid: "7724c6b6dd244c938163a023293ef5ea" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 39351 } }
I20260929 20:24:48.036154 29509 raft_consensus.cc:374] T 99126d56ba5f4e0890042aacc51d9e24 P 2414b15baffc47b7bdd1cd1db2e0d9d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:24:48.036262 29509 raft_consensus.cc:729] T 99126d56ba5f4e0890042aacc51d9e24 P 2414b15baffc47b7bdd1cd1db2e0d9d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2414b15baffc47b7bdd1cd1db2e0d9d2, State: Initialized, Role: FOLLOWER
I20260929 20:24:48.036900 29509 consensus_queue.cc:260] T 99126d56ba5f4e0890042aacc51d9e24 P 2414b15baffc47b7bdd1cd1db2e0d9d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2414b15baffc47b7bdd1cd1db2e0d9d2" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 39737 } } peers { permanent_uuid: "398106ca592f4051a04f795b1dadc73a" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 38875 } } peers { permanent_uuid: "7724c6b6dd244c938163a023293ef5ea" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 39351 } }
I20260929 20:24:48.038651 29509 ts_tablet_manager.cc:1428] T 99126d56ba5f4e0890042aacc51d9e24 P 2414b15baffc47b7bdd1cd1db2e0d9d2: Time spent starting tablet: real 0.040s user 0.015s sys 0.000s
W20260929 20:24:48.060738 29490 tablet.cc:2369] T 99126d56ba5f4e0890042aacc51d9e24 P 398106ca592f4051a04f795b1dadc73a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 20:24:48.093048 29356 tablet.cc:2369] T 99126d56ba5f4e0890042aacc51d9e24 P 7724c6b6dd244c938163a023293ef5ea: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 20:24:48.235306 29514 raft_consensus.cc:482] T 99126d56ba5f4e0890042aacc51d9e24 P 7724c6b6dd244c938163a023293ef5ea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 20:24:48.235657 29514 raft_consensus.cc:504] T 99126d56ba5f4e0890042aacc51d9e24 P 7724c6b6dd244c938163a023293ef5ea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2414b15baffc47b7bdd1cd1db2e0d9d2" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 39737 } } peers { permanent_uuid: "398106ca592f4051a04f795b1dadc73a" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 38875 } } peers { permanent_uuid: "7724c6b6dd244c938163a023293ef5ea" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 39351 } }
I20260929 20:24:48.237435 29514 leader_election.cc:288] T 99126d56ba5f4e0890042aacc51d9e24 P 7724c6b6dd244c938163a023293ef5ea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2414b15baffc47b7bdd1cd1db2e0d9d2 (127.17.53.65:39737), 398106ca592f4051a04f795b1dadc73a (127.17.53.67:38875)
I20260929 20:24:48.244762 29503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:24:48.238076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.53.66:53231 (local address 127.17.53.65:39737)
0929 20:24:48.238313 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0929 20:24:48.238319 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:24:48.239165 (+ 846us) server_negotiation.cc:407] Connection header received
0929 20:24:48.239516 (+ 351us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:24:48.239523 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:24:48.239609 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:24:48.239662 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:24:48.240349 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:24:48.241107 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:24:48.242958 (+ 1851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:24:48.243800 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:24:48.243989 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:24:48.244070 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0929 20:24:48.244340 (+ 270us) server_negotiation.cc:299] Negotiation successful
0929 20:24:48.244474 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260929 20:24:48.245204 29517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:24:48.237831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.65:39737 (local address 127.17.53.66:53231)
0929 20:24:48.239047 (+ 1216us) negotiation.cc:107] Waiting for socket to connect
0929 20:24:48.239066 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 20:24:48.239283 (+ 217us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:24:48.239882 (+ 599us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:24:48.239885 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:24:48.239924 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:24:48.240209 (+ 285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:24:48.240216 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:24:48.241247 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:24:48.241254 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:24:48.242781 (+ 1527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:24:48.242792 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:24:48.244021 (+ 1229us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:24:48.244027 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:24:48.244182 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:24:48.244205 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 20:24:48.244313 (+ 108us) client_negotiation.cc:240] Negotiation successful
0929 20:24:48.244447 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":922,"mutex_wait_us":50,"thread_start_us":588,"threads_started":1}
I20260929 20:24:48.245257 29507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:24:48.239348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.53.66:55401 (local address 127.17.53.67:38875)
0929 20:24:48.239553 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0929 20:24:48.239558 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:24:48.239790 (+ 232us) server_negotiation.cc:407] Connection header received
0929 20:24:48.239994 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:24:48.240000 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:24:48.240068 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:24:48.240170 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:24:48.240884 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:24:48.241594 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:24:48.243567 (+ 1973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:24:48.244343 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:24:48.244535 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:24:48.244609 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0929 20:24:48.244894 (+ 285us) server_negotiation.cc:299] Negotiation successful
0929 20:24:48.245019 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260929 20:24:48.245388 29518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:24:48.239050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.67:38875 (local address 127.17.53.66:55401)
0929 20:24:48.239680 (+ 630us) negotiation.cc:107] Waiting for socket to connect
0929 20:24:48.239698 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0929 20:24:48.239850 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:24:48.240198 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:24:48.240202 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:24:48.240226 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:24:48.240716 (+ 490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:24:48.240725 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:24:48.241736 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:24:48.241741 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:24:48.243417 (+ 1676us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:24:48.243428 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:24:48.244568 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:24:48.244574 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:24:48.244739 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:24:48.244762 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 20:24:48.244863 (+ 101us) client_negotiation.cc:240] Negotiation successful
0929 20:24:48.244999 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":518,"mutex_wait_us":28,"thread_start_us":331,"threads_started":1}
I20260929 20:24:48.246558 29444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99126d56ba5f4e0890042aacc51d9e24" candidate_uuid: "7724c6b6dd244c938163a023293ef5ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "398106ca592f4051a04f795b1dadc73a" is_pre_election: true
I20260929 20:24:48.246711 29167 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99126d56ba5f4e0890042aacc51d9e24" candidate_uuid: "7724c6b6dd244c938163a023293ef5ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2414b15baffc47b7bdd1cd1db2e0d9d2" is_pre_election: true
I20260929 20:24:48.247212 29444 raft_consensus.cc:2454] T 99126d56ba5f4e0890042aacc51d9e24 P 398106ca592f4051a04f795b1dadc73a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7724c6b6dd244c938163a023293ef5ea in term 0.
I20260929 20:24:48.247293 29167 raft_consensus.cc:2454] T 99126d56ba5f4e0890042aacc51d9e24 P 2414b15baffc47b7bdd1cd1db2e0d9d2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7724c6b6dd244c938163a023293ef5ea in term 0.
I20260929 20:24:48.247880 29240 leader_election.cc:302] T 99126d56ba5f4e0890042aacc51d9e24 P 7724c6b6dd244c938163a023293ef5ea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2414b15baffc47b7bdd1cd1db2e0d9d2, 7724c6b6dd244c938163a023293ef5ea; no voters:
I20260929 20:24:48.248310 29514 raft_consensus.cc:2781] T 99126d56ba5f4e0890042aacc51d9e24 P 7724c6b6dd244c938163a023293ef5ea [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 20:24:48.248468 29514 raft_consensus.cc:482] T 99126d56ba5f4e0890042aacc51d9e24 P 7724c6b6dd244c938163a023293ef5ea [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 20:24:48.248592 29514 raft_consensus.cc:3037] T 99126d56ba5f4e0890042aacc51d9e24 P 7724c6b6dd244c938163a023293ef5ea [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:24:48.251312 29514 raft_consensus.cc:504] T 99126d56ba5f4e0890042aacc51d9e24 P 7724c6b6dd244c938163a023293ef5ea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2414b15baffc47b7bdd1cd1db2e0d9d2" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 39737 } } peers { permanent_uuid: "398106ca592f4051a04f795b1dadc73a" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 38875 } } peers { permanent_uuid: "7724c6b6dd244c938163a023293ef5ea" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 39351 } }
I20260929 20:24:48.252059 29514 leader_election.cc:288] T 99126d56ba5f4e0890042aacc51d9e24 P 7724c6b6dd244c938163a023293ef5ea [CANDIDATE]: Term 1 election: Requested vote from peers 2414b15baffc47b7bdd1cd1db2e0d9d2 (127.17.53.65:39737), 398106ca592f4051a04f795b1dadc73a (127.17.53.67:38875)
I20260929 20:24:48.252506 29167 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99126d56ba5f4e0890042aacc51d9e24" candidate_uuid: "7724c6b6dd244c938163a023293ef5ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2414b15baffc47b7bdd1cd1db2e0d9d2"
I20260929 20:24:48.252718 29167 raft_consensus.cc:3037] T 99126d56ba5f4e0890042aacc51d9e24 P 2414b15baffc47b7bdd1cd1db2e0d9d2 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:24:48.252744 29444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99126d56ba5f4e0890042aacc51d9e24" candidate_uuid: "7724c6b6dd244c938163a023293ef5ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "398106ca592f4051a04f795b1dadc73a"
I20260929 20:24:48.253031 29444 raft_consensus.cc:3037] T 99126d56ba5f4e0890042aacc51d9e24 P 398106ca592f4051a04f795b1dadc73a [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:24:48.255322 29167 raft_consensus.cc:2454] T 99126d56ba5f4e0890042aacc51d9e24 P 2414b15baffc47b7bdd1cd1db2e0d9d2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7724c6b6dd244c938163a023293ef5ea in term 1.
I20260929 20:24:48.255748 29240 leader_election.cc:302] T 99126d56ba5f4e0890042aacc51d9e24 P 7724c6b6dd244c938163a023293ef5ea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2414b15baffc47b7bdd1cd1db2e0d9d2, 7724c6b6dd244c938163a023293ef5ea; no voters:
I20260929 20:24:48.256068 29514 raft_consensus.cc:2781] T 99126d56ba5f4e0890042aacc51d9e24 P 7724c6b6dd244c938163a023293ef5ea [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:24:48.256304 29444 raft_consensus.cc:2454] T 99126d56ba5f4e0890042aacc51d9e24 P 398106ca592f4051a04f795b1dadc73a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7724c6b6dd244c938163a023293ef5ea in term 1.
I20260929 20:24:48.256565 29514 raft_consensus.cc:686] T 99126d56ba5f4e0890042aacc51d9e24 P 7724c6b6dd244c938163a023293ef5ea [term 1 LEADER]: Becoming Leader. State: Replica: 7724c6b6dd244c938163a023293ef5ea, State: Running, Role: LEADER
I20260929 20:24:48.257368 29514 consensus_queue.cc:237] T 99126d56ba5f4e0890042aacc51d9e24 P 7724c6b6dd244c938163a023293ef5ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2414b15baffc47b7bdd1cd1db2e0d9d2" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 39737 } } peers { permanent_uuid: "398106ca592f4051a04f795b1dadc73a" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 38875 } } peers { permanent_uuid: "7724c6b6dd244c938163a023293ef5ea" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 39351 } }
I20260929 20:24:48.262295 28997 catalog_manager.cc:5495] T 99126d56ba5f4e0890042aacc51d9e24 P 7724c6b6dd244c938163a023293ef5ea reported cstate change: term changed from 0 to 1, leader changed from <none> to 7724c6b6dd244c938163a023293ef5ea (127.17.53.66). New cstate: current_term: 1 leader_uuid: "7724c6b6dd244c938163a023293ef5ea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2414b15baffc47b7bdd1cd1db2e0d9d2" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 39737 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "398106ca592f4051a04f795b1dadc73a" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 38875 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7724c6b6dd244c938163a023293ef5ea" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 39351 } health_report { overall_health: HEALTHY } } }
W20260929 20:24:48.278271 29213 tablet.cc:2369] T 99126d56ba5f4e0890042aacc51d9e24 P 2414b15baffc47b7bdd1cd1db2e0d9d2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 20:24:48.283077 29505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:24:48.277766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52718 (local address 127.17.53.66:39351)
0929 20:24:48.277982 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0929 20:24:48.277989 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:24:48.278056 (+ 67us) server_negotiation.cc:407] Connection header received
0929 20:24:48.278260 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:24:48.278264 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:24:48.278357 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:24:48.278548 (+ 191us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 20:24:48.278978 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:24:48.279693 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:24:48.281136 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:24:48.281523 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:24:48.281608 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:24:48.282029 (+ 421us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 20:24:48.282533 (+ 504us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 20:24:48.282638 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0929 20:24:48.282734 (+ 96us) server_negotiation.cc:299] Negotiation successful
0929 20:24:48.282797 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260929 20:24:48.291596 29444 raft_consensus.cc:1261] T 99126d56ba5f4e0890042aacc51d9e24 P 398106ca592f4051a04f795b1dadc73a [term 1 FOLLOWER]: Refusing update from remote peer 7724c6b6dd244c938163a023293ef5ea: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:24:48.291642 29167 raft_consensus.cc:1261] T 99126d56ba5f4e0890042aacc51d9e24 P 2414b15baffc47b7bdd1cd1db2e0d9d2 [term 1 FOLLOWER]: Refusing update from remote peer 7724c6b6dd244c938163a023293ef5ea: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:24:48.292557 29519 consensus_queue.cc:1035] T 99126d56ba5f4e0890042aacc51d9e24 P 7724c6b6dd244c938163a023293ef5ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "398106ca592f4051a04f795b1dadc73a" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 38875 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:24:48.292913 29514 consensus_queue.cc:1035] T 99126d56ba5f4e0890042aacc51d9e24 P 7724c6b6dd244c938163a023293ef5ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "2414b15baffc47b7bdd1cd1db2e0d9d2" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 39737 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:24:48.306145 29523 mvcc.cc:205] Tried to move back new op lower bound from 7334762448032628736 to 7334762447906349056. Current Snapshot: MvccSnapshot[applied={T|T < 7334762448032628736}]
I20260929 20:24:48.318516 28997 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45162:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 20:24:48.318804 28997 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45162:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 20:24:48.322547 28997 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0df11e4cac4e4955ac0307d6fe4614c0: Sending DeleteTablet for 3 replicas of tablet 99126d56ba5f4e0890042aacc51d9e24
I20260929 20:24:48.323743 29290 tablet_service.cc:1514] Processing DeleteTablet for tablet 99126d56ba5f4e0890042aacc51d9e24 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:24:48 UTC) from {username='kudu', principal='kudu/127.17.53.126@KRBTEST.COM'} at 127.0.0.1:52706
I20260929 20:24:48.323787 29146 tablet_service.cc:1514] Processing DeleteTablet for tablet 99126d56ba5f4e0890042aacc51d9e24 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:24:48 UTC) from {username='kudu', principal='kudu/127.17.53.126@KRBTEST.COM'} at 127.0.0.1:37944
I20260929 20:24:48.324378 29424 tablet_service.cc:1514] Processing DeleteTablet for tablet 99126d56ba5f4e0890042aacc51d9e24 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:24:48 UTC) from {username='kudu', principal='kudu/127.17.53.126@KRBTEST.COM'} at 127.0.0.1:57674
I20260929 20:24:48.326053 17621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu pid 29078
I20260929 20:24:48.347421 29532 tablet_replica.cc:330] T 99126d56ba5f4e0890042aacc51d9e24 P 2414b15baffc47b7bdd1cd1db2e0d9d2: stopping tablet replica
I20260929 20:24:48.348399 29533 tablet_replica.cc:330] T 99126d56ba5f4e0890042aacc51d9e24 P 398106ca592f4051a04f795b1dadc73a: stopping tablet replica
I20260929 20:24:48.348332 29532 raft_consensus.cc:2229] T 99126d56ba5f4e0890042aacc51d9e24 P 2414b15baffc47b7bdd1cd1db2e0d9d2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 20:24:48.348845 29533 raft_consensus.cc:2229] T 99126d56ba5f4e0890042aacc51d9e24 P 398106ca592f4051a04f795b1dadc73a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 20:24:48.349159 29532 raft_consensus.cc:2258] T 99126d56ba5f4e0890042aacc51d9e24 P 2414b15baffc47b7bdd1cd1db2e0d9d2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:24:48.349320 29533 raft_consensus.cc:2258] T 99126d56ba5f4e0890042aacc51d9e24 P 398106ca592f4051a04f795b1dadc73a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:24:48.350626 29534 tablet_replica.cc:330] T 99126d56ba5f4e0890042aacc51d9e24 P 7724c6b6dd244c938163a023293ef5ea: stopping tablet replica
I20260929 20:24:48.351119 29534 raft_consensus.cc:2229] T 99126d56ba5f4e0890042aacc51d9e24 P 7724c6b6dd244c938163a023293ef5ea [term 1 LEADER]: Raft consensus shutting down.
I20260929 20:24:48.351780 29532 ts_tablet_manager.cc:1905] T 99126d56ba5f4e0890042aacc51d9e24 P 2414b15baffc47b7bdd1cd1db2e0d9d2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:24:48.351915 29534 raft_consensus.cc:2258] T 99126d56ba5f4e0890042aacc51d9e24 P 7724c6b6dd244c938163a023293ef5ea [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:24:48.354838 29533 ts_tablet_manager.cc:1905] T 99126d56ba5f4e0890042aacc51d9e24 P 398106ca592f4051a04f795b1dadc73a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:24:48.355726 29534 ts_tablet_manager.cc:1905] T 99126d56ba5f4e0890042aacc51d9e24 P 7724c6b6dd244c938163a023293ef5ea: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 29 20:24:48 dist-test-slave-zbnc krb5kdc[28520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713483, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.53.65@KRBTEST.COM
I20260929 20:24:48.360642 29532 ts_tablet_manager.cc:1918] T 99126d56ba5f4e0890042aacc51d9e24 P 2414b15baffc47b7bdd1cd1db2e0d9d2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 20:24:48.360831 29532 log.cc:1196] T 99126d56ba5f4e0890042aacc51d9e24 P 2414b15baffc47b7bdd1cd1db2e0d9d2: Deleting WAL directory at /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790713339161492-17621-0/minicluster-data/ts-0/wal/wals/99126d56ba5f4e0890042aacc51d9e24
I20260929 20:24:48.361347 29532 ts_tablet_manager.cc:1939] T 99126d56ba5f4e0890042aacc51d9e24 P 2414b15baffc47b7bdd1cd1db2e0d9d2: Deleting consensus metadata
I20260929 20:24:48.362571 29533 ts_tablet_manager.cc:1918] T 99126d56ba5f4e0890042aacc51d9e24 P 398106ca592f4051a04f795b1dadc73a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 20:24:48.363323 28982 catalog_manager.cc:4841] TS 2414b15baffc47b7bdd1cd1db2e0d9d2 (127.17.53.65:39737): tablet 99126d56ba5f4e0890042aacc51d9e24 (table test-table [id=aae83d1fae864387888308fbba55bfd0]) successfully deleted
I20260929 20:24:48.371502 29534 ts_tablet_manager.cc:1918] T 99126d56ba5f4e0890042aacc51d9e24 P 7724c6b6dd244c938163a023293ef5ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 20:24:48.371768 29533 log.cc:1196] T 99126d56ba5f4e0890042aacc51d9e24 P 398106ca592f4051a04f795b1dadc73a: Deleting WAL directory at /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790713339161492-17621-0/minicluster-data/ts-2/wal/wals/99126d56ba5f4e0890042aacc51d9e24
I20260929 20:24:48.372337 29503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:24:48.329535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37958 (local address 127.17.53.65:39737)
0929 20:24:48.329778 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0929 20:24:48.329786 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 20:24:48.347722 (+ 17936us) server_negotiation.cc:407] Connection header received
0929 20:24:48.347943 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:24:48.347948 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:24:48.348029 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:24:48.348146 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:24:48.350282 (+ 2136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:24:48.351385 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:24:48.353440 (+ 2055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:24:48.354113 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:24:48.354218 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:24:48.364392 (+ 10174us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:24:48.364416 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:24:48.364433 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:24:48.364476 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:24:48.368467 (+ 3991us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:24:48.369479 (+ 1012us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:24:48.369490 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:24:48.369504 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:24:48.369772 (+ 268us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:24:48.370223 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:24:48.370228 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:24:48.370231 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:24:48.370733 (+ 502us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:24:48.371079 (+ 346us) server_negotiation.cc:1035] Waiting for connection context
0929 20:24:48.371637 (+ 558us) server_negotiation.cc:299] Negotiation successful
0929 20:24:48.371884 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260929 20:24:48.372386 29533 ts_tablet_manager.cc:1939] T 99126d56ba5f4e0890042aacc51d9e24 P 398106ca592f4051a04f795b1dadc73a: Deleting consensus metadata
I20260929 20:24:48.373935 29207 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:24:48.374346 28982 catalog_manager.cc:4841] TS 398106ca592f4051a04f795b1dadc73a (127.17.53.67:38875): tablet 99126d56ba5f4e0890042aacc51d9e24 (table test-table [id=aae83d1fae864387888308fbba55bfd0]) successfully deleted
I20260929 20:24:48.374828 29534 log.cc:1196] T 99126d56ba5f4e0890042aacc51d9e24 P 7724c6b6dd244c938163a023293ef5ea: Deleting WAL directory at /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790713339161492-17621-0/minicluster-data/ts-1/wal/wals/99126d56ba5f4e0890042aacc51d9e24
I20260929 20:24:48.375540 29534 ts_tablet_manager.cc:1939] T 99126d56ba5f4e0890042aacc51d9e24 P 7724c6b6dd244c938163a023293ef5ea: Deleting consensus metadata
I20260929 20:24:48.377602 28983 catalog_manager.cc:4841] TS 7724c6b6dd244c938163a023293ef5ea (127.17.53.66:39351): tablet 99126d56ba5f4e0890042aacc51d9e24 (table test-table [id=aae83d1fae864387888308fbba55bfd0]) successfully deleted
I20260929 20:24:48.487654 17621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu with pid 29078
I20260929 20:24:48.506642 17621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu pid 29218
Sep 29 20:24:48 dist-test-slave-zbnc krb5kdc[28520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713483, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.53.66@KRBTEST.COM
I20260929 20:24:48.524572 29505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:24:48.507722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52724 (local address 127.17.53.66:39351)
0929 20:24:48.507941 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0929 20:24:48.507947 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:24:48.507965 (+ 18us) server_negotiation.cc:407] Connection header received
0929 20:24:48.508035 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:24:48.508039 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:24:48.508099 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:24:48.508225 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:24:48.509511 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:24:48.510345 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:24:48.511946 (+ 1601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:24:48.512704 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:24:48.512850 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:24:48.517264 (+ 4414us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:24:48.517286 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:24:48.517309 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:24:48.517361 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:24:48.521471 (+ 4110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:24:48.522311 (+ 840us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:24:48.522331 (+ 20us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:24:48.522334 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:24:48.522526 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:24:48.522900 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:24:48.522906 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:24:48.522909 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:24:48.523356 (+ 447us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:24:48.523687 (+ 331us) server_negotiation.cc:1035] Waiting for connection context
0929 20:24:48.523937 (+ 250us) server_negotiation.cc:299] Negotiation successful
0929 20:24:48.524161 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260929 20:24:48.525309 29350 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:24:48.640937 17621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu with pid 29218
I20260929 20:24:48.656991 17621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu pid 29361
Sep 29 20:24:48 dist-test-slave-zbnc krb5kdc[28520](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713483, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.53.67@KRBTEST.COM
I20260929 20:24:48.671968 29507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:24:48.657756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57678 (local address 127.17.53.67:38875)
0929 20:24:48.657916 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0929 20:24:48.657922 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:24:48.657939 (+ 17us) server_negotiation.cc:407] Connection header received
0929 20:24:48.658084 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:24:48.658088 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:24:48.658147 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:24:48.658231 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:24:48.659660 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:24:48.660354 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:24:48.661576 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:24:48.661950 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:24:48.662034 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:24:48.666510 (+ 4476us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:24:48.666525 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:24:48.666541 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:24:48.666574 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:24:48.669541 (+ 2967us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:24:48.670215 (+ 674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:24:48.670222 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:24:48.670224 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:24:48.670346 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:24:48.670683 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:24:48.670686 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:24:48.670687 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:24:48.671114 (+ 427us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:24:48.671318 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0929 20:24:48.671554 (+ 236us) server_negotiation.cc:299] Negotiation successful
0929 20:24:48.671708 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260929 20:24:48.672624 29484 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:24:48.761976 17621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu with pid 29361
I20260929 20:24:48.778946 17621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu pid 28965
I20260929 20:24:48.780244 29027 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:24:48.893631 17621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu with pid 28965
I20260929 20:24:48.919324 17621 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260929 20:24:49.255201 17621 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260929 20:24:49.255505 17621 mini_ranger.cc:66] Stopping Ranger...
I20260929 20:24:49.601562 17621 mini_ranger.cc:68] Stopped Ranger
2026-09-29 20:24:49.601 UTC [28550] LOG: received smart shutdown request
2026-09-29 20:24:49.605 UTC [28550] LOG: background worker "logical replication launcher" (PID 28559) exited with exit code 1
2026-09-29 20:24:49.606 UTC [28554] LOG: shutting down
2026-09-29 20:24:49.749 UTC [28550] LOG: database system is shut down
2026-09-29T20:24:49Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (51219 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713339161492-17621-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 20:24:49 dist-test-slave-zbnc krb5kdc[29548](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 20:24:49 dist-test-slave-zbnc krb5kdc[29548](info): set up 2 sockets
Sep 29 20:24:49 dist-test-slave-zbnc krb5kdc[29548](info): commencing operation
krb5kdc: starting...
W20260929 20:24:50.988842 17621 mini_kdc.cc:121] Time spent starting KDC: real 1.074s user 0.000s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713339161492-17621-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713339161492-17621-0/krb5kdc/test-user.keytab.
Sep 29 20:24:51 dist-test-slave-zbnc krb5kdc[29548](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713491, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T20:24:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T20:24:51Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.17.53.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.17.53.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.17.53.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713339161492-17621-0/krb5kdc/rangeradmin_127.17.53.84@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.17.53.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713339161492-17621-0/krb5kdc/rangeradmin_127.17.53.84@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.17.53.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.17.53.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.17.53.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713339161492-17621-0/krb5kdc/rangerlookup_127.17.53.84@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.17.53.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713339161492-17621-0/krb5kdc/rangerlookup_127.17.53.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.17.53.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.17.53.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.17.53.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713339161492-17621-0/krb5kdc/HTTP_127.17.53.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.17.53.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713339161492-17621-0/krb5kdc/HTTP_127.17.53.84@KRBTEST.COM.keytab.
I20260929 20:24:51.322294 17621 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713339161492-17621-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskfEhy_t/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713339161492-17621-0/minicluster-data/postgres -l logfile start
2026-09-29 20:24:53.912 UTC [29578] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-29 20:24:53.912 UTC [29578] LOG: listening on IPv4 address "127.17.53.84", port 42947
2026-09-29 20:24:53.916 UTC [29578] LOG: listening on Unix socket "/tmp/.s.PGSQL.42947"
2026-09-29 20:24:53.938 UTC [29581] LOG: database system was shut down at 2026-09-29 20:24:52 UTC
2026-09-29 20:24:53.942 UTC [29578] LOG: database system is ready to accept connections
I20260929 20:24:54.916283 17621 mini_postgres.cc:96] Postgres bound to 42947
2026-09-29 20:24:54.938 UTC [29589] FATAL: database "slave" does not exist
127.17.53.84:42947 - accepting connections
I20260929 20:24:54.938829 17621 mini_ranger.cc:161] Starting Ranger...
I20260929 20:24:54.965754 17621 mini_ranger.cc:84] Created miniranger Postgres user
I20260929 20:24:55.494176 17621 mini_ranger.cc:87] Created ranger Postgres database
I20260929 20:24:55.494300 17621 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713339161492-17621-0/minicluster-data/ranger-admin
2026-09-29 20:24:55,814 [I] DB FLAVOR :POSTGRES
2026-09-29 20:24:55,815 [I] --------- Verifying Ranger DB connection ---------
2026-09-29 20:24:55,815 [I] Checking connection..
2026-09-29 20:24:55,815 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfEhy_t/build/asan/bin/postgresql.jar:/tmp/dist-test-taskfEhy_t/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.53.84:42947/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-29 20:24:56,047 [I] Checking connection passed.
2026-09-29 20:24:56,047 [I] --------- Verifying version history table ---------
2026-09-29 20:24:56,047 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfEhy_t/build/asan/bin/postgresql.jar:/tmp/dist-test-taskfEhy_t/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.53.84:42947/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 20:24:56,303 [I] Table x_db_version_h does not exist in database ranger
2026-09-29 20:24:56,304 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfEhy_t/build/asan/bin/postgresql.jar:/tmp/dist-test-taskfEhy_t/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.53.84:42947/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 20:24:56,562 [I] Table x_db_version_h does not exist in database ranger
2026-09-29 20:24:56,562 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-29 20:24:56,563 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfEhy_t/build/asan/bin/postgresql.jar:/tmp/dist-test-taskfEhy_t/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.53.84:42947/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskfEhy_t/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-29 20:24:56.820 UTC [29654] WARNING: there is no transaction in progress
2026-09-29 20:24:56,826 [I] create_dbversion_catalog.sql file imported successfully
2026-09-29 20:24:56,827 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfEhy_t/build/asan/bin/postgresql.jar:/tmp/dist-test-taskfEhy_t/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.53.84:42947/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-29 20:24:57,059 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-29 20:24:57,059 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-29 20:24:57,060 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfEhy_t/build/asan/bin/postgresql.jar:/tmp/dist-test-taskfEhy_t/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.53.84:42947/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-29 20:24:57,335 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfEhy_t/build/asan/bin/postgresql.jar:/tmp/dist-test-taskfEhy_t/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.53.84:42947/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-29 20:24:57,583 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfEhy_t/build/asan/bin/postgresql.jar:/tmp/dist-test-taskfEhy_t/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.53.84:42947/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-zbnc.c.gcp-upstream.internal','N') ;"
2026-09-29 20:24:57,809 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfEhy_t/build/asan/bin/postgresql.jar:/tmp/dist-test-taskfEhy_t/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.53.84:42947/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-29 20:24:58,042 [I] Table x_portal_user does not exist in database ranger
2026-09-29 20:24:58,042 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfEhy_t/build/asan/bin/postgresql.jar:/tmp/dist-test-taskfEhy_t/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.53.84:42947/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-29 20:24:58,289 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-29 20:24:58,289 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfEhy_t/build/asan/bin/postgresql.jar:/tmp/dist-test-taskfEhy_t/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.53.84:42947/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-29 20:24:58,518 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfEhy_t/build/asan/bin/postgresql.jar:/tmp/dist-test-taskfEhy_t/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.53.84:42947/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-29 20:24:58,751 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-29 20:24:58,751 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfEhy_t/build/asan/bin/postgresql.jar:/tmp/dist-test-taskfEhy_t/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.53.84:42947/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskfEhy_t/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-29 20:24:59.727 UTC [29763] WARNING: there is no transaction in progress
2026-09-29 20:24:59.745 UTC [29763] WARNING: there is no transaction in progress
2026-09-29 20:24:59.762 UTC [29763] WARNING: there is no transaction in progress
2026-09-29 20:24:59.777 UTC [29763] WARNING: there is no transaction in progress
2026-09-29 20:24:59.794 UTC [29763] WARNING: there is no transaction in progress
2026-09-29 20:24:59.815 UTC [29763] WARNING: there is no transaction in progress
2026-09-29 20:24:59.875 UTC [29763] WARNING: there is no transaction in progress
2026-09-29 20:24:59.881 UTC [29763] WARNING: there is no transaction in progress
2026-09-29 20:24:59.896 UTC [29763] WARNING: there is no transaction in progress
2026-09-29 20:24:59.907 UTC [29763] WARNING: there is no transaction in progress
2026-09-29 20:24:59.917 UTC [29763] WARNING: there is no transaction in progress
2026-09-29 20:24:59.932 UTC [29763] WARNING: there is no transaction in progress
2026-09-29 20:24:59.942 UTC [29763] WARNING: there is no transaction in progress
2026-09-29 20:24:59.949 UTC [29763] WARNING: there is no transaction in progress
2026-09-29 20:24:59.956 UTC [29763] WARNING: there is no transaction in progress
2026-09-29 20:24:59.963 UTC [29763] WARNING: there is no transaction in progress
2026-09-29 20:25:00,627 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-29 20:25:00,627 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfEhy_t/build/asan/bin/postgresql.jar:/tmp/dist-test-taskfEhy_t/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.53.84:42947/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-29 20:25:00,881 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-29 20:25:00,881 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfEhy_t/build/asan/bin/postgresql.jar:/tmp/dist-test-taskfEhy_t/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.53.84:42947/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-29 20:25:01,136 [I] Table x_portal_user already exists in database 'ranger'
2026-09-29 20:25:01,136 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfEhy_t/build/asan/bin/postgresql.jar:/tmp/dist-test-taskfEhy_t/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.53.84:42947/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-29 20:25:01,373 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-29 20:25:01,373 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfEhy_t/build/asan/bin/postgresql.jar:/tmp/dist-test-taskfEhy_t/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.53.84:42947/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-29 20:25:01,618 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfEhy_t/build/asan/bin/postgresql.jar:/tmp/dist-test-taskfEhy_t/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.53.84:42947/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-29 20:25:01,891 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfEhy_t/build/asan/bin/postgresql.jar:/tmp/dist-test-taskfEhy_t/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.53.84:42947/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-zbnc.c.gcp-upstream.internal';"
2026-09-29 20:25:02,132 [I] CORE_DB_SCHEMA import status has been updated
2026-09-29 20:25:02,133 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfEhy_t/build/asan/bin/postgresql.jar:/tmp/dist-test-taskfEhy_t/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.53.84:42947/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-29 20:25:02,379 [I] DB_PATCHES have already been applied
I20260929 20:25:02.385417 17621 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713339161492-17621-0/minicluster-data/ranger-admin:/tmp/dist-test-taskfEhy_t/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskfEhy_t/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskfEhy_t/build/asan/bin/postgresql.jar:/tmp/dist-test-taskfEhy_t/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260929 20:25:02.385510 17621 mini_ranger.cc:193] Using host: 127.17.53.84
I20260929 20:25:02.400643 17621 mini_ranger.cc:227] Ranger admin URL: http://127.17.53.84:58217
Sep 29, 2026 8:25:02 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 29, 2026 8:25:02 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 29, 2026 8:25:02 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713339161492-17621-0/minicluster-data/ranger-admin/ews/webapp
Sep 29, 2026 8:25:02 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713339161492-17621-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 29, 2026 8:25:02 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713339161492-17621-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 29, 2026 8:25:03 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713339161492-17621-0/minicluster-data/ranger-admin/ews/webapp].
Sep 29, 2026 8:25:03 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260929 20:25:03.517951 17621 mini_ranger.cc:160] Time spent starting Ranger: real 8.579s user 0.009s sys 0.032s
Sep 29, 2026 8:25:24 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 29, 2026 8:25:24 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 29, 2026 8:25:24 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 29, 2026 8:25:24 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 29, 2026 8:25:24 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 29, 2026 8:25:24 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 29, 2026 8:25:24 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 29, 2026 8:25:24 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 29, 2026 8:25:24 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 29, 2026 8:25:24 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 29, 2026 8:25:24 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 29, 2026 8:25:24 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 29, 2026 8:25:24 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 29, 2026 8:25:24 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 29, 2026 8:25:24 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 29, 2026 8:25:24 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 29, 2026 8:25:24 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 29, 2026 8:25:24 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260929 20:25:25.573484 17621 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.17.53.84@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.17.53.84@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.17.53.84@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713339161492-17621-0/krb5kdc/rangerkms_127.17.53.84@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.17.53.84@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713339161492-17621-0/krb5kdc/rangerkms_127.17.53.84@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.17.53.84@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.17.53.84@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.17.53.84@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713339161492-17621-0/krb5kdc/HTTP_127.17.53.84@KRBTEST.COM.keytab.
Entry for principal HTTP/127.17.53.84@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713339161492-17621-0/krb5kdc/HTTP_127.17.53.84@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 29 20:25:25 dist-test-slave-zbnc krb5kdc[29548](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713525, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260929 20:25:25.742798 17621 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260929 20:25:25.763232 17621 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260929 20:25:26.723529 17621 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260929 20:25:26.723647 17621 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713339161492-17621-0/minicluster-data/ranger-kms
I20260929 20:25:26.723694 17621 mini_ranger_kms.cc:227] Using postgres at 127.17.53.84:42947
2026-09-29 20:25:27,154 [I] DB FLAVOR :POSTGRES
2026-09-29 20:25:27,155 [I] --------- Verifying Ranger DB connection ---------
2026-09-29 20:25:27,155 [I] Checking connection
2026-09-29 20:25:27,155 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfEhy_t/build/asan/bin/postgresql.jar:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713339161492-17621-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.53.84:42947/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-29 20:25:27,405 [I] connection success
2026-09-29 20:25:27,406 [I] --------- Verifying Ranger DB tables ---------
2026-09-29 20:25:27,406 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-29 20:25:27,406 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfEhy_t/build/asan/bin/postgresql.jar:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713339161492-17621-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.53.84:42947/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-29 20:25:27,656 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-29 20:25:27,657 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-29 20:25:27,657 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-29 20:25:27,657 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfEhy_t/build/asan/bin/postgresql.jar:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713339161492-17621-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.17.53.84:42947/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713339161492-17621-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-29 20:25:27,919 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260929 20:25:28.377120 17621 mini_ranger_kms.cc:310] Created kms service
I20260929 20:25:28.545609 17621 mini_ranger_kms.cc:326] Created kudu user
I20260929 20:25:28.590257 17621 mini_ranger_kms.cc:343] Created rangerkms user
Sep 29 20:25:28 dist-test-slave-zbnc krb5kdc[29548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713525, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.17.53.84@KRBTEST.COM
I20260929 20:25:28.957940 17621 mini_ranger_kms.cc:384] Added ranger policy
I20260929 20:25:28.958252 17621 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713339161492-17621-0/minicluster-data/ranger-kms:/tmp/dist-test-taskfEhy_t/build/asan/bin/postgresql.jar:/tmp/dist-test-taskfEhy_t/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskfEhy_t/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260929 20:25:28.958299 17621 mini_ranger_kms.cc:240] Using host: 127.17.53.84
I20260929 20:25:28.976848 17621 mini_ranger_kms.cc:276] Ranger KMS PID: 29943
I20260929 20:25:28.976992 17621 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.17.53.84:58281
Sep 29, 2026 8:25:30 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 29, 2026 8:25:30 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 29, 2026 8:25:30 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713339161492-17621-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 29, 2026 8:25:30 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713339161492-17621-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 29, 2026 8:25:30 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713339161492-17621-0/minicluster-data/ranger-kms/ews/webapp].
Sep 29, 2026 8:25:30 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Sep 29 20:25:31 dist-test-slave-zbnc krb5kdc[29548](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790713531, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.17.53.84@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:25:31.383185 17621 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.640s user 0.069s sys 0.327s
I20260929 20:25:31.383438 17621 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260929 20:25:31.383620 17621 mini_ranger_kms.cc:401] 127.17.53.84:58281/kms/v1/keys
Sep 29 20:25:31 dist-test-slave-zbnc krb5kdc[29548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713531, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.17.53.84@KRBTEST.COM for HTTP/127.17.53.84@KRBTEST.COM
Sep 29 20:25:32 dist-test-slave-zbnc krb5kdc[29548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713531, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.17.53.84@KRBTEST.COM for HTTP/127.17.53.84@KRBTEST.COM
Sep 29 20:25:34 dist-test-slave-zbnc krb5kdc[29548](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713534, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.17.53.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.53.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.53.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.126.keytab.
Entry for principal kudu/127.17.53.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.126.keytab.
I20260929 20:25:34.342383 17621 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
/tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713339161492-17621-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713339161492-17621-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713339161492-17621-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713339161492-17621-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.53.126:46595
--webserver_interface=127.17.53.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.126.keytab
--principal=kudu/127.17.53.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.53.84:39811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.53.126:46595
--ranger_config_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713339161492-17621-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.17.53.84:58281/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713339161492-17621-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713339161492-17621-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713339161492-17621-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:25:34.676900 29994 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:25:34.677230 29994 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:25:34.677399 29994 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:25:34.684850 29994 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:25:34.684960 29994 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:25:34.685063 29994 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:25:34.685155 29994 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:25:34.685240 29994 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:25:34.695514 29994 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.53.84:39811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713339161492-17621-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713339161492-17621-0/minicluster-data/master-0/wal
--ranger_kms_url=127.17.53.84:58281/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.17.53.126:46595
--ranger_config_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713339161492-17621-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.126.keytab
--principal=kudu/127.17.53.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.53.126:46595
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713339161492-17621-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.53.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713339161492-17621-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:25:34.696777 29994 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:25:34.698271 29994 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:25:34.710623 29999 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:25:34.710549 30002 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:25:34.710588 30000 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:25:34.710922 29994 server_base.cc:1034] running on GCE node
I20260929 20:25:34.712083 29994 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:25:34.713635 29994 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:25:34.714903 29994 hybrid_clock.cc:648] HybridClock initialized: now 1790713534714822 us; error 36 us; skew 500 ppm
I20260929 20:25:34.715243 29994 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:25:34 dist-test-slave-zbnc krb5kdc[29548](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713534, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.53.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:25:34.718253 29994 init.cc:377] Logged in from keytab as kudu/127.17.53.126@KRBTEST.COM (short username kudu)
I20260929 20:25:34.719732 29994 webserver.cc:460] Webserver started at http://127.17.53.126:35663/ using document root <none> and password file <none>
I20260929 20:25:34.720376 29994 fs_manager.cc:359] Metadata directory not provided
I20260929 20:25:34.720507 29994 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:25:34.720814 29994 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 20:25:34 dist-test-slave-zbnc krb5kdc[29548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713534, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.53.126@KRBTEST.COM for HTTP/127.17.53.84@KRBTEST.COM
I20260929 20:25:34.783458 29994 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713339161492-17621-0/minicluster-data/master-0/data/instance:
uuid: "0f40088f621e4863961f6a9da4daee82"
format_stamp: "Formatted at 2026-09-29 20:25:34 on dist-test-slave-zbnc"
server_key: "35da8651956f324eae017e3878ecf4df"
server_key_iv: "cc74102d5a8ffc666eb17f5cb23b2c2c"
server_key_version: "kuduclusterkey@0"
I20260929 20:25:34.784289 29994 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713339161492-17621-0/minicluster-data/master-0/wal/instance:
uuid: "0f40088f621e4863961f6a9da4daee82"
format_stamp: "Formatted at 2026-09-29 20:25:34 on dist-test-slave-zbnc"
server_key: "35da8651956f324eae017e3878ecf4df"
server_key_iv: "cc74102d5a8ffc666eb17f5cb23b2c2c"
server_key_version: "kuduclusterkey@0"
I20260929 20:25:34.789597 29994 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260929 20:25:34.808676 30009 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:25:34.810460 29994 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260929 20:25:34.810673 29994 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713339161492-17621-0/minicluster-data/master-0/data,/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713339161492-17621-0/minicluster-data/master-0/wal
uuid: "0f40088f621e4863961f6a9da4daee82"
format_stamp: "Formatted at 2026-09-29 20:25:34 on dist-test-slave-zbnc"
server_key: "35da8651956f324eae017e3878ecf4df"
server_key_iv: "cc74102d5a8ffc666eb17f5cb23b2c2c"
server_key_version: "kuduclusterkey@0"
I20260929 20:25:34.810973 29994 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713339161492-17621-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713339161492-17621-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713339161492-17621-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:25:34.837045 29994 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:25:34.840731 29994 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:25:34.841006 29994 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:25:34.859556 29994 rpc_server.cc:307] RPC server started. Bound to: 127.17.53.126:46595
I20260929 20:25:34.859577 30060 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.53.126:46595 every 8 connection(s)
I20260929 20:25:34.860961 29994 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713339161492-17621-0/minicluster-data/master-0/data/info.pb
I20260929 20:25:34.864734 30061 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:25:34.866176 17621 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu as pid 29994
I20260929 20:25:34.866403 17621 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713339161492-17621-0/minicluster-data/master-0/wal/instance
I20260929 20:25:34.867308 17621 external_mini_cluster.cc:1428] Setting key 1ff0ac7bbf451864842b541252c6def5
I20260929 20:25:34.877555 30061 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0f40088f621e4863961f6a9da4daee82: Bootstrap starting.
Sep 29 20:25:34 dist-test-slave-zbnc krb5kdc[29548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713534, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.53.126@KRBTEST.COM
I20260929 20:25:34.880748 30061 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0f40088f621e4863961f6a9da4daee82: Neither blocks nor log segments found. Creating new log.
I20260929 20:25:34.881752 30061 log.cc:824] T 00000000000000000000000000000000 P 0f40088f621e4863961f6a9da4daee82: Log is configured to *not* fsync() on all Append() calls
I20260929 20:25:34.884750 30061 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0f40088f621e4863961f6a9da4daee82: No bootstrap required, opened a new log
I20260929 20:25:34.888669 30064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:25:34.869470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36974 (local address 127.17.53.126:46595)
0929 20:25:34.870261 (+ 791us) server_negotiation.cc:206] Beginning negotiation
0929 20:25:34.870276 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 20:25:34.870325 (+ 49us) server_negotiation.cc:407] Connection header received
0929 20:25:34.871912 (+ 1587us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:25:34.871936 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:25:34.872315 (+ 379us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:25:34.872676 (+ 361us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:25:34.874017 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:25:34.875381 (+ 1364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:25:34.876688 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:25:34.877347 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:25:34.877548 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:25:34.881270 (+ 3722us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:25:34.881277 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:25:34.881294 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:25:34.881332 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:25:34.884315 (+ 2983us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:25:34.885001 (+ 686us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:25:34.885009 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:25:34.885017 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:25:34.885230 (+ 213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:25:34.885580 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:25:34.885585 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:25:34.885589 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:25:34.886237 (+ 648us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:25:34.886552 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0929 20:25:34.886881 (+ 329us) server_negotiation.cc:299] Negotiation successful
0929 20:25:34.887296 (+ 415us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":594,"thread_start_us":383,"threads_started":1}
I20260929 20:25:34.894104 30061 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0f40088f621e4863961f6a9da4daee82 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f40088f621e4863961f6a9da4daee82" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 46595 } }
I20260929 20:25:34.894544 30061 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0f40088f621e4863961f6a9da4daee82 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:25:34.894658 30061 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0f40088f621e4863961f6a9da4daee82 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f40088f621e4863961f6a9da4daee82, State: Initialized, Role: FOLLOWER
I20260929 20:25:34.895255 30061 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0f40088f621e4863961f6a9da4daee82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f40088f621e4863961f6a9da4daee82" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 46595 } }
I20260929 20:25:34.895486 30061 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0f40088f621e4863961f6a9da4daee82 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 20:25:34.895622 30061 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0f40088f621e4863961f6a9da4daee82 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 20:25:34.895810 30061 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0f40088f621e4863961f6a9da4daee82 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:25:34.897877 30061 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0f40088f621e4863961f6a9da4daee82 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f40088f621e4863961f6a9da4daee82" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 46595 } }
I20260929 20:25:34.898346 30061 leader_election.cc:302] T 00000000000000000000000000000000 P 0f40088f621e4863961f6a9da4daee82 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0f40088f621e4863961f6a9da4daee82; no voters:
I20260929 20:25:34.899004 30061 leader_election.cc:288] T 00000000000000000000000000000000 P 0f40088f621e4863961f6a9da4daee82 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 20:25:34.899214 30066 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0f40088f621e4863961f6a9da4daee82 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:25:34.899901 30066 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0f40088f621e4863961f6a9da4daee82 [term 1 LEADER]: Becoming Leader. State: Replica: 0f40088f621e4863961f6a9da4daee82, State: Running, Role: LEADER
I20260929 20:25:34.900475 30066 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0f40088f621e4863961f6a9da4daee82 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f40088f621e4863961f6a9da4daee82" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 46595 } }
I20260929 20:25:34.901386 30061 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0f40088f621e4863961f6a9da4daee82 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:25:34.903294 30067 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f40088f621e4863961f6a9da4daee82 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0f40088f621e4863961f6a9da4daee82" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f40088f621e4863961f6a9da4daee82" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 46595 } } }
I20260929 20:25:34.903725 30067 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f40088f621e4863961f6a9da4daee82 [sys.catalog]: This master's current role is: LEADER
I20260929 20:25:34.906239 30072 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:25:34.903293 30068 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f40088f621e4863961f6a9da4daee82 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0f40088f621e4863961f6a9da4daee82. Latest consensus state: current_term: 1 leader_uuid: "0f40088f621e4863961f6a9da4daee82" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f40088f621e4863961f6a9da4daee82" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 46595 } } }
I20260929 20:25:34.906970 30068 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f40088f621e4863961f6a9da4daee82 [sys.catalog]: This master's current role is: LEADER
I20260929 20:25:34.911285 30072 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:25:34.912031 30061 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713339161492-17621-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260929 20:25:34.929838 30072 catalog_manager.cc:1347] Generated new cluster ID: 5e63164c6cbb43f3bbf07133ecf2f778
I20260929 20:25:34.929986 30072 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:25:34.949738 30072 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 20:25:34.951843 30072 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:25:34.961550 30072 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0f40088f621e4863961f6a9da4daee82: Generated new TSK 0
I20260929 20:25:34.962689 30072 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.17.53.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.53.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.53.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.65.keytab.
Entry for principal kudu/127.17.53.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.65.keytab.
I20260929 20:25:35.718048 17621 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
/tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713339161492-17621-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713339161492-17621-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713339161492-17621-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713339161492-17621-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.17.53.84:58281/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.53.65:0
--local_ip_for_outbound_sockets=127.17.53.65
--webserver_interface=127.17.53.65
--webserver_port=0
--tserver_master_addrs=127.17.53.126:46595
--keytab_file=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.65.keytab
--principal=kudu/127.17.53.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.53.84:39811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713339161492-17621-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713339161492-17621-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713339161492-17621-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 20:25:35 dist-test-slave-zbnc krb5kdc[29548](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790713535, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.53.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260929 20:25:36.176450 30106 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:25:36.176795 30106 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:25:36.176940 30106 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:25:36.184432 30106 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:25:36.184553 30106 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:25:36.184728 30106 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.53.65
I20260929 20:25:36.194809 30106 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.53.84:39811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713339161492-17621-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713339161492-17621-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.17.53.84:58281/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.65.keytab
--principal=kudu/127.17.53.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.53.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713339161492-17621-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.17.53.65
--webserver_port=0
--tserver_master_addrs=127.17.53.126:46595
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.53.65
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713339161492-17621-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:25:36.196231 30106 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:25:36.197613 30106 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:25:36.224133 30116 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:25:36.226079 30117 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:25:36.235158 30119 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:25:36.236946 30106 server_base.cc:1034] running on GCE node
I20260929 20:25:36.238191 30106 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:25:36.241309 30106 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:25:36.244845 30106 hybrid_clock.cc:648] HybridClock initialized: now 1790713536244714 us; error 111 us; skew 500 ppm
I20260929 20:25:36.245358 30106 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:25:36 dist-test-slave-zbnc krb5kdc[29548](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713536, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.53.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:25:36.250517 30106 init.cc:377] Logged in from keytab as kudu/127.17.53.65@KRBTEST.COM (short username kudu)
I20260929 20:25:36.254371 30106 webserver.cc:460] Webserver started at http://127.17.53.65:36035/ using document root <none> and password file <none>
I20260929 20:25:36.255421 30106 fs_manager.cc:359] Metadata directory not provided
I20260929 20:25:36.255573 30106 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:25:36.256126 30106 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 20:25:36 dist-test-slave-zbnc krb5kdc[29548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713536, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.53.65@KRBTEST.COM for HTTP/127.17.53.84@KRBTEST.COM
I20260929 20:25:36.281242 30106 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713339161492-17621-0/minicluster-data/ts-0/data/instance:
uuid: "7b4c9e1d97384eaa8121341ae228d059"
format_stamp: "Formatted at 2026-09-29 20:25:36 on dist-test-slave-zbnc"
server_key: "23adc0392f36a40091e3ef461a62bcc5"
server_key_iv: "c27361856d779a96c8157f4be69ff2b3"
server_key_version: "kuduclusterkey@0"
I20260929 20:25:36.282516 30106 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713339161492-17621-0/minicluster-data/ts-0/wal/instance:
uuid: "7b4c9e1d97384eaa8121341ae228d059"
format_stamp: "Formatted at 2026-09-29 20:25:36 on dist-test-slave-zbnc"
server_key: "23adc0392f36a40091e3ef461a62bcc5"
server_key_iv: "c27361856d779a96c8157f4be69ff2b3"
server_key_version: "kuduclusterkey@0"
I20260929 20:25:36.298041 30106 fs_manager.cc:691] Time spent creating directory manager: real 0.015s user 0.007s sys 0.007s
I20260929 20:25:36.338590 30126 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:25:36.341318 30106 fs_manager.cc:725] Time spent opening block manager: real 0.016s user 0.009s sys 0.004s
I20260929 20:25:36.341625 30106 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713339161492-17621-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713339161492-17621-0/minicluster-data/ts-0/wal
uuid: "7b4c9e1d97384eaa8121341ae228d059"
format_stamp: "Formatted at 2026-09-29 20:25:36 on dist-test-slave-zbnc"
server_key: "23adc0392f36a40091e3ef461a62bcc5"
server_key_iv: "c27361856d779a96c8157f4be69ff2b3"
server_key_version: "kuduclusterkey@0"
I20260929 20:25:36.341980 30106 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713339161492-17621-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713339161492-17621-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713339161492-17621-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
Sep 29 20:25:36 dist-test-slave-zbnc krb5kdc[29548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713535, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.53.126@KRBTEST.COM for HTTP/127.17.53.84@KRBTEST.COM
I20260929 20:25:36.408260 30106 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:25:36.413473 30106 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:25:36.413864 30106 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:25:36.421568 30106 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:25:36.423702 30106 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:25:36.423837 30106 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:25:36.424036 30106 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:25:36.424103 30106 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 29 20:25:36 dist-test-slave-zbnc krb5kdc[29548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713535, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.53.126@KRBTEST.COM for HTTP/127.17.53.84@KRBTEST.COM
I20260929 20:25:36.968978 30106 rpc_server.cc:307] RPC server started. Bound to: 127.17.53.65:33033
I20260929 20:25:36.969074 30242 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.53.65:33033 every 8 connection(s)
I20260929 20:25:36.971107 30106 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713339161492-17621-0/minicluster-data/ts-0/data/info.pb
I20260929 20:25:36.974189 17621 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu as pid 30106
I20260929 20:25:36.974426 17621 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713339161492-17621-0/minicluster-data/ts-0/wal/instance
I20260929 20:25:36.975301 17621 external_mini_cluster.cc:1428] Setting key 0987ea13051c8e2abbc9c56c304896ef
Sep 29 20:25:36 dist-test-slave-zbnc krb5kdc[29548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713536, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.53.65@KRBTEST.COM for kudu/127.17.53.126@KRBTEST.COM
I20260929 20:25:37.000854 30246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:25:36.975937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:46595 (local address 127.17.53.65:43905)
0929 20:25:36.979405 (+ 3468us) negotiation.cc:107] Waiting for socket to connect
0929 20:25:36.979455 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0929 20:25:36.981171 (+ 1716us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:25:36.981950 (+ 779us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:25:36.981958 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:25:36.982420 (+ 462us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:25:36.983083 (+ 663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:25:36.983103 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:25:36.984390 (+ 1287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:25:36.984395 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:25:36.986216 (+ 1821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:25:36.986231 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:25:36.987153 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:25:36.987163 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:25:36.987488 (+ 325us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:25:36.988359 (+ 871us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:25:36.988384 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:25:36.991054 (+ 2670us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:25:36.995789 (+ 4735us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:25:36.995813 (+ 24us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:25:36.995831 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:25:36.996496 (+ 665us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:25:36.996977 (+ 481us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:25:36.996982 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:25:36.996985 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:25:36.997268 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:25:36.998017 (+ 749us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:25:36.998025 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:25:36.998477 (+ 452us) client_negotiation.cc:769] Sending connection context
0929 20:25:36.998838 (+ 361us) client_negotiation.cc:240] Negotiation successful
0929 20:25:36.999386 (+ 548us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":563,"spinlock_wait_cycles":1793280,"thread_start_us":519,"threads_started":1}
I20260929 20:25:37.002003 30245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:25:36.975274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.53.65:43905 (local address 127.17.53.126:46595)
0929 20:25:36.979320 (+ 4046us) server_negotiation.cc:206] Beginning negotiation
0929 20:25:36.979326 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:25:36.979569 (+ 243us) server_negotiation.cc:407] Connection header received
0929 20:25:36.981449 (+ 1880us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:25:36.981454 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:25:36.981553 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:25:36.981674 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:25:36.983259 (+ 1585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:25:36.984215 (+ 956us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:25:36.986389 (+ 2174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:25:36.986906 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:25:36.987034 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:25:36.991761 (+ 4727us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:25:36.991770 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:25:36.991779 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:25:36.991838 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:25:36.995615 (+ 3777us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:25:36.996652 (+ 1037us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:25:36.996658 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:25:36.996664 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:25:36.996834 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:25:36.997416 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:25:36.997421 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:25:36.997424 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:25:36.997780 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:25:36.997941 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0929 20:25:37.001392 (+ 3451us) server_negotiation.cc:299] Negotiation successful
0929 20:25:37.001618 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3916,"thread_start_us":3777,"threads_started":1}
I20260929 20:25:37.003791 30243 heartbeater.cc:347] Connected to a master server at 127.17.53.126:46595
I20260929 20:25:37.004263 30243 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.17.53.66@KRBTEST.COM; defaulting to no policy
I20260929 20:25:37.005264 30243 heartbeater.cc:511] Master 127.17.53.126:46595 requested a full tablet report, sending...
I20260929 20:25:37.008144 30026 ts_manager.cc:194] Registered new tserver with Master: 7b4c9e1d97384eaa8121341ae228d059 (127.17.53.65:33033)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.53.66@KRBTEST.COM" created.
I20260929 20:25:37.010221 30026 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.53.65@KRBTEST.COM'} at 127.17.53.65:43905
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.53.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.66.keytab.
Entry for principal kudu/127.17.53.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.66.keytab.
I20260929 20:25:37.062805 17621 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
/tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713339161492-17621-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713339161492-17621-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713339161492-17621-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713339161492-17621-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.17.53.84:58281/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.53.66:0
--local_ip_for_outbound_sockets=127.17.53.66
--webserver_interface=127.17.53.66
--webserver_port=0
--tserver_master_addrs=127.17.53.126:46595
--keytab_file=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.66.keytab
--principal=kudu/127.17.53.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.53.84:39811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713339161492-17621-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713339161492-17621-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713339161492-17621-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:25:37.429081 30249 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:25:37.429447 30249 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:25:37.429549 30249 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:25:37.437143 30249 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:25:37.437270 30249 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:25:37.437386 30249 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.53.66
I20260929 20:25:37.447582 30249 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.53.84:39811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713339161492-17621-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713339161492-17621-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.17.53.84:58281/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.66.keytab
--principal=kudu/127.17.53.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.53.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713339161492-17621-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.17.53.66
--webserver_port=0
--tserver_master_addrs=127.17.53.126:46595
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.53.66
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713339161492-17621-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:25:37.449431 30249 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:25:37.450795 30249 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:25:37.464733 30261 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:25:37.464779 30264 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:25:37.464710 30262 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:25:37.465673 30249 server_base.cc:1034] running on GCE node
I20260929 20:25:37.466290 30249 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:25:37.467437 30249 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:25:37.468685 30249 hybrid_clock.cc:648] HybridClock initialized: now 1790713537468615 us; error 69 us; skew 500 ppm
I20260929 20:25:37.469067 30249 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:25:37 dist-test-slave-zbnc krb5kdc[29548](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713537, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.53.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:25:37.477108 30249 init.cc:377] Logged in from keytab as kudu/127.17.53.66@KRBTEST.COM (short username kudu)
I20260929 20:25:37.478693 30249 webserver.cc:460] Webserver started at http://127.17.53.66:34809/ using document root <none> and password file <none>
I20260929 20:25:37.479553 30249 fs_manager.cc:359] Metadata directory not provided
I20260929 20:25:37.479665 30249 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:25:37.479935 30249 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 20:25:37 dist-test-slave-zbnc krb5kdc[29548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713537, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.53.66@KRBTEST.COM for HTTP/127.17.53.84@KRBTEST.COM
I20260929 20:25:37.496395 30249 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713339161492-17621-0/minicluster-data/ts-1/data/instance:
uuid: "6701d4dcd11647d9801a1e2ef14e2bcd"
format_stamp: "Formatted at 2026-09-29 20:25:37 on dist-test-slave-zbnc"
server_key: "2bd3eb700de2c85ff0fb3ff5b2834c4d"
server_key_iv: "9c36d20fa41e65f417068ae62c4b4634"
server_key_version: "kuduclusterkey@0"
I20260929 20:25:37.497308 30249 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713339161492-17621-0/minicluster-data/ts-1/wal/instance:
uuid: "6701d4dcd11647d9801a1e2ef14e2bcd"
format_stamp: "Formatted at 2026-09-29 20:25:37 on dist-test-slave-zbnc"
server_key: "2bd3eb700de2c85ff0fb3ff5b2834c4d"
server_key_iv: "9c36d20fa41e65f417068ae62c4b4634"
server_key_version: "kuduclusterkey@0"
I20260929 20:25:37.503404 30249 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260929 20:25:37.522279 30271 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:25:37.524024 30249 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.000s
I20260929 20:25:37.524237 30249 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713339161492-17621-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713339161492-17621-0/minicluster-data/ts-1/wal
uuid: "6701d4dcd11647d9801a1e2ef14e2bcd"
format_stamp: "Formatted at 2026-09-29 20:25:37 on dist-test-slave-zbnc"
server_key: "2bd3eb700de2c85ff0fb3ff5b2834c4d"
server_key_iv: "9c36d20fa41e65f417068ae62c4b4634"
server_key_version: "kuduclusterkey@0"
I20260929 20:25:37.524495 30249 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713339161492-17621-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713339161492-17621-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713339161492-17621-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:25:37.549147 30249 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:25:37.553094 30249 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:25:37.553385 30249 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:25:37.554298 30249 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:25:37.555828 30249 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:25:37.555903 30249 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:25:37.555987 30249 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:25:37.556031 30249 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:25:37.589944 30249 rpc_server.cc:307] RPC server started. Bound to: 127.17.53.66:43573
I20260929 20:25:37.589984 30383 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.53.66:43573 every 8 connection(s)
I20260929 20:25:37.591372 30249 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713339161492-17621-0/minicluster-data/ts-1/data/info.pb
I20260929 20:25:37.596401 17621 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu as pid 30249
I20260929 20:25:37.596596 17621 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713339161492-17621-0/minicluster-data/ts-1/wal/instance
I20260929 20:25:37.597028 17621 external_mini_cluster.cc:1428] Setting key 01f9c15a27c8e275dad115df98a96667
Sep 29 20:25:37 dist-test-slave-zbnc krb5kdc[29548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713537, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.53.66@KRBTEST.COM for kudu/127.17.53.126@KRBTEST.COM
I20260929 20:25:37.614156 30386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:25:37.593977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.53.66:59393 (local address 127.17.53.126:46595)
0929 20:25:37.594437 (+ 460us) server_negotiation.cc:206] Beginning negotiation
0929 20:25:37.594442 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:25:37.595421 (+ 979us) server_negotiation.cc:407] Connection header received
0929 20:25:37.597373 (+ 1952us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:25:37.597381 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:25:37.597480 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:25:37.597610 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:25:37.599427 (+ 1817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:25:37.600222 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:25:37.602848 (+ 2626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:25:37.603378 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:25:37.603477 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:25:37.607715 (+ 4238us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:25:37.607726 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:25:37.607729 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:25:37.607762 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:25:37.610982 (+ 3220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:25:37.611761 (+ 779us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:25:37.611767 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:25:37.611771 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:25:37.611899 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:25:37.612331 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:25:37.612336 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:25:37.612339 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:25:37.612626 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:25:37.612780 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0929 20:25:37.613545 (+ 765us) server_negotiation.cc:299] Negotiation successful
0929 20:25:37.613773 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":357,"thread_start_us":243,"threads_started":1}
I20260929 20:25:37.615260 30387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:25:37.594410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:46595 (local address 127.17.53.66:59393)
0929 20:25:37.595220 (+ 810us) negotiation.cc:107] Waiting for socket to connect
0929 20:25:37.595292 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0929 20:25:37.597064 (+ 1772us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:25:37.597838 (+ 774us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:25:37.597847 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:25:37.598416 (+ 569us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:25:37.599237 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:25:37.599264 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:25:37.600391 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:25:37.600399 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:25:37.602693 (+ 2294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:25:37.602712 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:25:37.603606 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:25:37.603615 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:25:37.603966 (+ 351us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:25:37.604770 (+ 804us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:25:37.604819 (+ 49us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:25:37.607480 (+ 2661us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:25:37.611143 (+ 3663us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:25:37.611158 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:25:37.611173 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:25:37.611622 (+ 449us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:25:37.612015 (+ 393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:25:37.612020 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:25:37.612025 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:25:37.612205 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:25:37.612756 (+ 551us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:25:37.612773 (+ 17us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:25:37.613152 (+ 379us) client_negotiation.cc:769] Sending connection context
0929 20:25:37.613448 (+ 296us) client_negotiation.cc:240] Negotiation successful
0929 20:25:37.613999 (+ 551us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":553,"thread_start_us":293,"threads_started":1}
I20260929 20:25:37.616922 30384 heartbeater.cc:347] Connected to a master server at 127.17.53.126:46595
I20260929 20:25:37.617264 30384 heartbeater.cc:464] Registering TS with master...
I20260929 20:25:37.618541 30384 heartbeater.cc:511] Master 127.17.53.126:46595 requested a full tablet report, sending...
I20260929 20:25:37.620398 30026 ts_manager.cc:194] Registered new tserver with Master: 6701d4dcd11647d9801a1e2ef14e2bcd (127.17.53.66:43573)
I20260929 20:25:37.621204 30026 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.53.66@KRBTEST.COM'} at 127.17.53.66:59393
WARNING: no policy specified for kudu/127.17.53.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.53.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.53.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.67.keytab.
Entry for principal kudu/127.17.53.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.67.keytab.
I20260929 20:25:37.654671 17621 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
/tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713339161492-17621-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713339161492-17621-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713339161492-17621-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713339161492-17621-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.17.53.84:58281/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.53.67:0
--local_ip_for_outbound_sockets=127.17.53.67
--webserver_interface=127.17.53.67
--webserver_port=0
--tserver_master_addrs=127.17.53.126:46595
--keytab_file=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.67.keytab
--principal=kudu/127.17.53.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.53.84:39811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713339161492-17621-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713339161492-17621-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713339161492-17621-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:25:37.987442 30390 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:25:37.987738 30390 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:25:37.987833 30390 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:25:37.995095 30390 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:25:37.995201 30390 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:25:37.995409 30390 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.53.67
I20260929 20:25:38.005439 30390 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.53.84:39811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713339161492-17621-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713339161492-17621-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.17.53.84:58281/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.67.keytab
--principal=kudu/127.17.53.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.53.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713339161492-17621-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.17.53.67
--webserver_port=0
--tserver_master_addrs=127.17.53.126:46595
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.53.67
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713339161492-17621-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:25:38.006749 30390 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:25:38.008198 30390 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 20:25:38.018277 30243 heartbeater.cc:503] Master 127.17.53.126:46595 was elected leader, sending a full tablet report...
W20260929 20:25:38.021875 30396 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:25:38.022095 30398 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:25:38.022486 30395 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:25:38.023541 30390 server_base.cc:1034] running on GCE node
I20260929 20:25:38.024204 30390 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:25:38.025207 30390 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:25:38.026403 30390 hybrid_clock.cc:648] HybridClock initialized: now 1790713538026357 us; error 33 us; skew 500 ppm
I20260929 20:25:38.026739 30390 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:25:38 dist-test-slave-zbnc krb5kdc[29548](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713538, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.53.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:25:38.030176 30390 init.cc:377] Logged in from keytab as kudu/127.17.53.67@KRBTEST.COM (short username kudu)
I20260929 20:25:38.031744 30390 webserver.cc:460] Webserver started at http://127.17.53.67:43685/ using document root <none> and password file <none>
I20260929 20:25:38.032418 30390 fs_manager.cc:359] Metadata directory not provided
I20260929 20:25:38.032516 30390 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:25:38.032778 30390 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 29 20:25:38 dist-test-slave-zbnc krb5kdc[29548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713538, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.53.67@KRBTEST.COM for HTTP/127.17.53.84@KRBTEST.COM
I20260929 20:25:38.046277 30390 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713339161492-17621-0/minicluster-data/ts-2/data/instance:
uuid: "3cd60b52e8654707a72d84c66e8a31cb"
format_stamp: "Formatted at 2026-09-29 20:25:38 on dist-test-slave-zbnc"
server_key: "c75d4942ec6ff83e70877a4afc3e7fb5"
server_key_iv: "628ebfae33a18a360c67fe960aee90f0"
server_key_version: "kuduclusterkey@0"
I20260929 20:25:38.047223 30390 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713339161492-17621-0/minicluster-data/ts-2/wal/instance:
uuid: "3cd60b52e8654707a72d84c66e8a31cb"
format_stamp: "Formatted at 2026-09-29 20:25:38 on dist-test-slave-zbnc"
server_key: "c75d4942ec6ff83e70877a4afc3e7fb5"
server_key_iv: "628ebfae33a18a360c67fe960aee90f0"
server_key_version: "kuduclusterkey@0"
I20260929 20:25:38.052932 30390 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260929 20:25:38.067837 30405 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:25:38.069438 30390 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260929 20:25:38.069633 30390 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713339161492-17621-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713339161492-17621-0/minicluster-data/ts-2/wal
uuid: "3cd60b52e8654707a72d84c66e8a31cb"
format_stamp: "Formatted at 2026-09-29 20:25:38 on dist-test-slave-zbnc"
server_key: "c75d4942ec6ff83e70877a4afc3e7fb5"
server_key_iv: "628ebfae33a18a360c67fe960aee90f0"
server_key_version: "kuduclusterkey@0"
I20260929 20:25:38.069909 30390 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713339161492-17621-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713339161492-17621-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713339161492-17621-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:25:38.099277 30390 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:25:38.103586 30390 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:25:38.103948 30390 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:25:38.104997 30390 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:25:38.106562 30390 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:25:38.106679 30390 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:25:38.106843 30390 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:25:38.106940 30390 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260929 20:25:38.145471 30390 rpc_server.cc:307] RPC server started. Bound to: 127.17.53.67:46175
I20260929 20:25:38.145507 30517 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.53.67:46175 every 8 connection(s)
I20260929 20:25:38.146951 30390 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713339161492-17621-0/minicluster-data/ts-2/data/info.pb
I20260929 20:25:38.148506 17621 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu as pid 30390
I20260929 20:25:38.148712 17621 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713339161492-17621-0/minicluster-data/ts-2/wal/instance
I20260929 20:25:38.149266 17621 external_mini_cluster.cc:1428] Setting key ed776368c645d2145aad5060d614559f
Sep 29 20:25:38 dist-test-slave-zbnc krb5kdc[29548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713538, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.53.67@KRBTEST.COM for kudu/127.17.53.126@KRBTEST.COM
I20260929 20:25:38.167330 30520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:25:38.149928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.53.67:55263 (local address 127.17.53.126:46595)
0929 20:25:38.150413 (+ 485us) server_negotiation.cc:206] Beginning negotiation
0929 20:25:38.150418 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:25:38.151648 (+ 1230us) server_negotiation.cc:407] Connection header received
0929 20:25:38.153447 (+ 1799us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:25:38.153451 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:25:38.153542 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:25:38.153639 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:25:38.155067 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:25:38.155783 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:25:38.157318 (+ 1535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:25:38.157759 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:25:38.157863 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:25:38.161322 (+ 3459us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:25:38.161328 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:25:38.161332 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:25:38.161366 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:25:38.164382 (+ 3016us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:25:38.165145 (+ 763us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:25:38.165150 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:25:38.165154 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:25:38.165318 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:25:38.165729 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:25:38.165733 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:25:38.165735 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:25:38.166014 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:25:38.166124 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0929 20:25:38.166816 (+ 692us) server_negotiation.cc:299] Negotiation successful
0929 20:25:38.167072 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":367,"thread_start_us":241,"threads_started":1}
I20260929 20:25:38.168365 30521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:25:38.150602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:46595 (local address 127.17.53.67:55263)
0929 20:25:38.151499 (+ 897us) negotiation.cc:107] Waiting for socket to connect
0929 20:25:38.151551 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0929 20:25:38.153200 (+ 1649us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:25:38.153835 (+ 635us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:25:38.153858 (+ 23us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:25:38.154274 (+ 416us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:25:38.154852 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:25:38.154923 (+ 71us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:25:38.155925 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:25:38.155929 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:25:38.157180 (+ 1251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:25:38.157191 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:25:38.157977 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:25:38.157982 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:25:38.158191 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:25:38.158968 (+ 777us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:25:38.158992 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:25:38.161069 (+ 2077us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:25:38.164542 (+ 3473us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:25:38.164552 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:25:38.164571 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:25:38.165051 (+ 480us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:25:38.165450 (+ 399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:25:38.165453 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:25:38.165455 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:25:38.165630 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:25:38.166124 (+ 494us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:25:38.166130 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:25:38.166476 (+ 346us) client_negotiation.cc:769] Sending connection context
0929 20:25:38.166706 (+ 230us) client_negotiation.cc:240] Negotiation successful
0929 20:25:38.167182 (+ 476us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":624,"thread_start_us":387,"threads_started":1}
I20260929 20:25:38.169891 30518 heartbeater.cc:347] Connected to a master server at 127.17.53.126:46595
I20260929 20:25:38.170203 30518 heartbeater.cc:464] Registering TS with master...
I20260929 20:25:38.171015 30518 heartbeater.cc:511] Master 127.17.53.126:46595 requested a full tablet report, sending...
I20260929 20:25:38.172538 30026 ts_manager.cc:194] Registered new tserver with Master: 3cd60b52e8654707a72d84c66e8a31cb (127.17.53.67:46175)
I20260929 20:25:38.173410 30026 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.53.67@KRBTEST.COM'} at 127.17.53.67:55263
I20260929 20:25:38.177826 17621 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 20:25:38.179579 17621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu pid 30106
Sep 29 20:25:38 dist-test-slave-zbnc krb5kdc[29548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713534, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.53.65@KRBTEST.COM
I20260929 20:25:38.196823 30523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:25:38.180430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50300 (local address 127.17.53.65:33033)
0929 20:25:38.181044 (+ 614us) server_negotiation.cc:206] Beginning negotiation
0929 20:25:38.181054 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 20:25:38.181152 (+ 98us) server_negotiation.cc:407] Connection header received
0929 20:25:38.181365 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:25:38.181375 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:25:38.181610 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:25:38.181794 (+ 184us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:25:38.183371 (+ 1577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:25:38.184418 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:25:38.185786 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:25:38.186382 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:25:38.186526 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:25:38.190787 (+ 4261us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:25:38.190799 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:25:38.190814 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:25:38.190902 (+ 88us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:25:38.193692 (+ 2790us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:25:38.194526 (+ 834us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:25:38.194535 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:25:38.194540 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:25:38.194682 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:25:38.195133 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:25:38.195139 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:25:38.195143 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:25:38.195504 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:25:38.195857 (+ 353us) server_negotiation.cc:1035] Waiting for connection context
0929 20:25:38.196166 (+ 309us) server_negotiation.cc:299] Negotiation successful
0929 20:25:38.196444 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":440,"thread_start_us":261,"threads_started":1}
I20260929 20:25:38.198136 30238 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:25:38.283236 17621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu with pid 30106
I20260929 20:25:38.299803 17621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu pid 30249
Sep 29 20:25:38 dist-test-slave-zbnc krb5kdc[29548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713534, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.53.66@KRBTEST.COM
I20260929 20:25:38.316706 30525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:25:38.300736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53146 (local address 127.17.53.66:43573)
0929 20:25:38.301298 (+ 562us) server_negotiation.cc:206] Beginning negotiation
0929 20:25:38.301305 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:25:38.301333 (+ 28us) server_negotiation.cc:407] Connection header received
0929 20:25:38.301398 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:25:38.301404 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:25:38.301631 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:25:38.301783 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:25:38.303256 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:25:38.304220 (+ 964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:25:38.305606 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:25:38.306166 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:25:38.306261 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:25:38.310701 (+ 4440us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:25:38.310715 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:25:38.310732 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:25:38.310769 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:25:38.313721 (+ 2952us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:25:38.314429 (+ 708us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:25:38.314436 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:25:38.314438 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:25:38.314560 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:25:38.315207 (+ 647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:25:38.315210 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:25:38.315212 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:25:38.315560 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:25:38.315822 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0929 20:25:38.316069 (+ 247us) server_negotiation.cc:299] Negotiation successful
0929 20:25:38.316318 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":409,"thread_start_us":318,"threads_started":1}
I20260929 20:25:38.318019 30379 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:25:38.400261 17621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu with pid 30249
I20260929 20:25:38.416354 17621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu pid 30390
Sep 29 20:25:38 dist-test-slave-zbnc krb5kdc[29548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713534, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.53.67@KRBTEST.COM
I20260929 20:25:38.433380 30527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:25:38.417315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37060 (local address 127.17.53.67:46175)
0929 20:25:38.417895 (+ 580us) server_negotiation.cc:206] Beginning negotiation
0929 20:25:38.417903 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 20:25:38.417931 (+ 28us) server_negotiation.cc:407] Connection header received
0929 20:25:38.417990 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:25:38.417996 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:25:38.418209 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:25:38.418386 (+ 177us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:25:38.419789 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:25:38.420787 (+ 998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:25:38.422307 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:25:38.423016 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:25:38.423151 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:25:38.427644 (+ 4493us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:25:38.427656 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:25:38.427671 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:25:38.427703 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:25:38.430424 (+ 2721us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:25:38.431210 (+ 786us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:25:38.431218 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:25:38.431220 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:25:38.431345 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:25:38.431729 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:25:38.431733 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:25:38.431735 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:25:38.432062 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:25:38.432399 (+ 337us) server_negotiation.cc:1035] Waiting for connection context
0929 20:25:38.432719 (+ 320us) server_negotiation.cc:299] Negotiation successful
0929 20:25:38.433027 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":438,"thread_start_us":273,"threads_started":1}
I20260929 20:25:38.434607 30513 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:25:38.516734 17621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu with pid 30390
I20260929 20:25:38.532436 17621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu pid 29994
I20260929 20:25:38.533761 30056 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:25:38.639871 17621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu with pid 29994
I20260929 20:25:38.661779 17621 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
/tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713339161492-17621-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713339161492-17621-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713339161492-17621-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713339161492-17621-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.53.126:46595
--webserver_interface=127.17.53.126
--webserver_port=35663
--keytab_file=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.126.keytab
--principal=kudu/127.17.53.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.53.84:39811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.53.126:46595
--ranger_config_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713339161492-17621-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.17.53.84:58281/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.17.53.84:58281/kms with env {KRB5CCNAME=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713339161492-17621-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713339161492-17621-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713339161492-17621-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:25:38.985592 30530 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:25:38.985878 30530 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:25:38.986027 30530 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:25:38.993193 30530 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:25:38.993301 30530 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:25:38.993386 30530 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:25:38.993479 30530 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:25:38.993568 30530 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:25:39.003635 30530 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.53.84:39811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713339161492-17621-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713339161492-17621-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.17.53.84:58281/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.17.53.126:46595
--ranger_config_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713339161492-17621-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.126.keytab
--principal=kudu/127.17.53.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.53.126:46595
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713339161492-17621-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.53.126
--webserver_port=35663
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713339161492-17621-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:25:39.004997 30530 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:25:39.006376 30530 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:25:39.018462 30535 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:25:39.018510 30536 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:25:39.018425 30538 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:25:39.024751 30530 server_base.cc:1034] running on GCE node
I20260929 20:25:39.025520 30530 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:25:39.027036 30530 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:25:39.028273 30530 hybrid_clock.cc:648] HybridClock initialized: now 1790713539028237 us; error 48 us; skew 500 ppm
I20260929 20:25:39.028644 30530 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:25:39 dist-test-slave-zbnc krb5kdc[29548](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713539, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.53.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:25:39.031949 30530 init.cc:377] Logged in from keytab as kudu/127.17.53.126@KRBTEST.COM (short username kudu)
I20260929 20:25:39.033690 30530 webserver.cc:460] Webserver started at http://127.17.53.126:35663/ using document root <none> and password file <none>
I20260929 20:25:39.034688 30530 fs_manager.cc:359] Metadata directory not provided
I20260929 20:25:39.034898 30530 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 29 20:25:39 dist-test-slave-zbnc krb5kdc[29548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713539, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.53.126@KRBTEST.COM for HTTP/127.17.53.84@KRBTEST.COM
I20260929 20:25:39.087088 30530 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 20:25:39.090340 30546 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:25:39.092044 30530 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 20:25:39.092262 30530 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713339161492-17621-0/minicluster-data/master-0/data,/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713339161492-17621-0/minicluster-data/master-0/wal
uuid: "0f40088f621e4863961f6a9da4daee82"
format_stamp: "Formatted at 2026-09-29 20:25:34 on dist-test-slave-zbnc"
server_key: "35da8651956f324eae017e3878ecf4df"
server_key_iv: "cc74102d5a8ffc666eb17f5cb23b2c2c"
server_key_version: "kuduclusterkey@0"
I20260929 20:25:39.093297 30530 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713339161492-17621-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713339161492-17621-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713339161492-17621-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:25:39.111517 30530 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:25:39.115523 30530 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:25:39.115832 30530 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:25:39.135951 30530 rpc_server.cc:307] RPC server started. Bound to: 127.17.53.126:46595
I20260929 20:25:39.136015 30597 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.53.126:46595 every 8 connection(s)
I20260929 20:25:39.138049 30530 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713339161492-17621-0/minicluster-data/master-0/data/info.pb
I20260929 20:25:39.145103 30598 sys_catalog.cc:263] Verifying existing consensus state
I20260929 20:25:39.145648 17621 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu as pid 30530
I20260929 20:25:39.146569 17621 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
/tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713339161492-17621-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713339161492-17621-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713339161492-17621-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713339161492-17621-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.17.53.84:58281/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.53.65:33033
--local_ip_for_outbound_sockets=127.17.53.65
--tserver_master_addrs=127.17.53.126:46595
--webserver_port=36035
--webserver_interface=127.17.53.65
--keytab_file=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.65.keytab
--principal=kudu/127.17.53.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.53.84:39811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.17.53.84:58281/kms with env {KRB5CCNAME=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713339161492-17621-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713339161492-17621-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713339161492-17621-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 20:25:39.148712 30598 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0f40088f621e4863961f6a9da4daee82: Bootstrap starting.
I20260929 20:25:39.167985 30598 log.cc:824] T 00000000000000000000000000000000 P 0f40088f621e4863961f6a9da4daee82: Log is configured to *not* fsync() on all Append() calls
I20260929 20:25:39.177187 30598 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0f40088f621e4863961f6a9da4daee82: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 20:25:39.177731 30598 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0f40088f621e4863961f6a9da4daee82: Bootstrap complete.
I20260929 20:25:39.187076 30598 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0f40088f621e4863961f6a9da4daee82 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f40088f621e4863961f6a9da4daee82" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 46595 } }
I20260929 20:25:39.188498 30598 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0f40088f621e4863961f6a9da4daee82 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f40088f621e4863961f6a9da4daee82, State: Initialized, Role: FOLLOWER
I20260929 20:25:39.189123 30598 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0f40088f621e4863961f6a9da4daee82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f40088f621e4863961f6a9da4daee82" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 46595 } }
I20260929 20:25:39.189373 30598 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0f40088f621e4863961f6a9da4daee82 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 20:25:39.189496 30598 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0f40088f621e4863961f6a9da4daee82 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 20:25:39.189687 30598 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0f40088f621e4863961f6a9da4daee82 [term 1 FOLLOWER]: Advancing to term 2
I20260929 20:25:39.193629 30598 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0f40088f621e4863961f6a9da4daee82 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f40088f621e4863961f6a9da4daee82" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 46595 } }
I20260929 20:25:39.194128 30598 leader_election.cc:302] T 00000000000000000000000000000000 P 0f40088f621e4863961f6a9da4daee82 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0f40088f621e4863961f6a9da4daee82; no voters:
I20260929 20:25:39.194787 30598 leader_election.cc:288] T 00000000000000000000000000000000 P 0f40088f621e4863961f6a9da4daee82 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260929 20:25:39.194952 30602 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0f40088f621e4863961f6a9da4daee82 [term 2 FOLLOWER]: Leader election won for term 2
I20260929 20:25:39.196324 30602 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0f40088f621e4863961f6a9da4daee82 [term 2 LEADER]: Becoming Leader. State: Replica: 0f40088f621e4863961f6a9da4daee82, State: Running, Role: LEADER
I20260929 20:25:39.196969 30602 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0f40088f621e4863961f6a9da4daee82 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f40088f621e4863961f6a9da4daee82" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 46595 } }
I20260929 20:25:39.197232 30598 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0f40088f621e4863961f6a9da4daee82 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:25:39.204375 30603 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f40088f621e4863961f6a9da4daee82 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "0f40088f621e4863961f6a9da4daee82" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f40088f621e4863961f6a9da4daee82" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 46595 } } }
I20260929 20:25:39.204794 30603 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f40088f621e4863961f6a9da4daee82 [sys.catalog]: This master's current role is: LEADER
I20260929 20:25:39.205498 30604 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0f40088f621e4863961f6a9da4daee82 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0f40088f621e4863961f6a9da4daee82. Latest consensus state: current_term: 2 leader_uuid: "0f40088f621e4863961f6a9da4daee82" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0f40088f621e4863961f6a9da4daee82" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 46595 } } }
I20260929 20:25:39.206256 30604 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0f40088f621e4863961f6a9da4daee82 [sys.catalog]: This master's current role is: LEADER
I20260929 20:25:39.215948 30608 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:25:39.222218 30608 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:25:39.224398 30608 catalog_manager.cc:1259] Loaded cluster ID: 5e63164c6cbb43f3bbf07133ecf2f778
I20260929 20:25:39.224588 30608 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:25:39.228658 30608 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:25:39.230151 30608 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 0f40088f621e4863961f6a9da4daee82: Loaded TSK: 0
I20260929 20:25:39.231963 30608 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 20:25:39.240270 30598 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713339161492-17621-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
W20260929 20:25:39.562489 30600 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:25:39.563017 30600 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:25:39.563252 30600 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:25:39.575290 30600 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:25:39.575518 30600 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:25:39.575748 30600 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.53.65
I20260929 20:25:39.592259 30600 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.53.84:39811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713339161492-17621-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713339161492-17621-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.17.53.84:58281/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.65.keytab
--principal=kudu/127.17.53.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.53.65:33033
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713339161492-17621-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.17.53.65
--webserver_port=36035
--tserver_master_addrs=127.17.53.126:46595
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.53.65
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713339161492-17621-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:25:39.594340 30600 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:25:39.596243 30600 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:25:39.615115 30636 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:25:39.615187 30635 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:25:39.628875 30638 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:25:39.628997 30600 server_base.cc:1034] running on GCE node
I20260929 20:25:39.631917 30600 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:25:39.633299 30600 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:25:39.636737 30600 hybrid_clock.cc:648] HybridClock initialized: now 1790713539636445 us; error 290 us; skew 500 ppm
I20260929 20:25:39.637213 30600 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:25:39 dist-test-slave-zbnc krb5kdc[29548](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713539, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.53.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:25:39.641803 30600 init.cc:377] Logged in from keytab as kudu/127.17.53.65@KRBTEST.COM (short username kudu)
I20260929 20:25:39.652025 30600 webserver.cc:460] Webserver started at http://127.17.53.65:36035/ using document root <none> and password file <none>
I20260929 20:25:39.653234 30600 fs_manager.cc:359] Metadata directory not provided
I20260929 20:25:39.653481 30600 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 29 20:25:39 dist-test-slave-zbnc krb5kdc[29548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713539, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.53.65@KRBTEST.COM for HTTP/127.17.53.84@KRBTEST.COM
I20260929 20:25:39.700518 30600 fs_manager.cc:709] Time spent opening directory manager: real 0.014s user 0.006s sys 0.005s
I20260929 20:25:39.711014 30646 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:25:39.713265 30600 fs_manager.cc:725] Time spent opening block manager: real 0.011s user 0.005s sys 0.005s
I20260929 20:25:39.713541 30600 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713339161492-17621-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713339161492-17621-0/minicluster-data/ts-0/wal
uuid: "7b4c9e1d97384eaa8121341ae228d059"
format_stamp: "Formatted at 2026-09-29 20:25:36 on dist-test-slave-zbnc"
server_key: "23adc0392f36a40091e3ef461a62bcc5"
server_key_iv: "c27361856d779a96c8157f4be69ff2b3"
server_key_version: "kuduclusterkey@0"
I20260929 20:25:39.715216 30600 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713339161492-17621-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713339161492-17621-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713339161492-17621-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:25:39.772939 30600 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:25:39.778302 30600 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:25:39.778680 30600 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:25:39.786171 30600 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:25:39.794234 30600 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:25:39.794430 30600 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260929 20:25:39.794602 30600 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:25:39.794744 30600 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:25:39.871923 30600 rpc_server.cc:307] RPC server started. Bound to: 127.17.53.65:33033
I20260929 20:25:39.871991 30760 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.53.65:33033 every 8 connection(s)
I20260929 20:25:39.873948 30600 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713339161492-17621-0/minicluster-data/ts-0/data/info.pb
I20260929 20:25:39.877748 17621 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu as pid 30600
I20260929 20:25:39.878602 17621 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
/tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713339161492-17621-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713339161492-17621-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713339161492-17621-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713339161492-17621-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.17.53.84:58281/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.53.66:43573
--local_ip_for_outbound_sockets=127.17.53.66
--tserver_master_addrs=127.17.53.126:46595
--webserver_port=34809
--webserver_interface=127.17.53.66
--keytab_file=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.66.keytab
--principal=kudu/127.17.53.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.53.84:39811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.17.53.84:58281/kms with env {KRB5CCNAME=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713339161492-17621-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713339161492-17621-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713339161492-17621-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 20:25:39 dist-test-slave-zbnc krb5kdc[29548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713539, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.53.65@KRBTEST.COM for kudu/127.17.53.126@KRBTEST.COM
I20260929 20:25:39.937811 30763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:25:39.877523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:46595 (local address 127.17.53.65:40929)
0929 20:25:39.888156 (+ 10633us) negotiation.cc:107] Waiting for socket to connect
0929 20:25:39.888220 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0929 20:25:39.893454 (+ 5234us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:25:39.904646 (+ 11192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:25:39.904727 (+ 81us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:25:39.905468 (+ 741us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:25:39.906414 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:25:39.906445 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:25:39.910713 (+ 4268us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:25:39.910722 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:25:39.913250 (+ 2528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:25:39.913272 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:25:39.915235 (+ 1963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:25:39.915242 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:25:39.915519 (+ 277us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:25:39.916353 (+ 834us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:25:39.916384 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:25:39.919232 (+ 2848us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:25:39.925092 (+ 5860us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:25:39.925101 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:25:39.925118 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:25:39.926984 (+ 1866us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:25:39.929241 (+ 2257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:25:39.929253 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:25:39.929256 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:25:39.930836 (+ 1580us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:25:39.935423 (+ 4587us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:25:39.935430 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:25:39.935807 (+ 377us) client_negotiation.cc:769] Sending connection context
0929 20:25:39.936094 (+ 287us) client_negotiation.cc:240] Negotiation successful
0929 20:25:39.936635 (+ 541us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10299,"thread_start_us":10058,"threads_started":1}
I20260929 20:25:39.940297 30764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:25:39.879438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.53.65:40929 (local address 127.17.53.126:46595)
0929 20:25:39.887480 (+ 8042us) server_negotiation.cc:206] Beginning negotiation
0929 20:25:39.887504 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0929 20:25:39.888397 (+ 893us) server_negotiation.cc:407] Connection header received
0929 20:25:39.898998 (+ 10601us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:25:39.899035 (+ 37us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:25:39.900494 (+ 1459us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:25:39.900949 (+ 455us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:25:39.907952 (+ 7003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:25:39.910418 (+ 2466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:25:39.913483 (+ 3065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:25:39.914611 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:25:39.914800 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:25:39.919587 (+ 4787us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:25:39.919598 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:25:39.919619 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:25:39.919677 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:25:39.924896 (+ 5219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:25:39.927173 (+ 2277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:25:39.927182 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:25:39.927190 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:25:39.927391 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:25:39.931035 (+ 3644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:25:39.931040 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:25:39.931043 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:25:39.935073 (+ 4030us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:25:39.935352 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0929 20:25:39.938658 (+ 3306us) server_negotiation.cc:299] Negotiation successful
0929 20:25:39.939101 (+ 443us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7748,"thread_start_us":7488,"threads_started":1}
I20260929 20:25:39.944042 30761 heartbeater.cc:347] Connected to a master server at 127.17.53.126:46595
I20260929 20:25:39.944367 30761 heartbeater.cc:464] Registering TS with master...
I20260929 20:25:39.945250 30761 heartbeater.cc:511] Master 127.17.53.126:46595 requested a full tablet report, sending...
W20260929 20:25:39.948513 30761 heartbeater.cc:644] Failed to heartbeat to 127.17.53.126:46595 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260929 20:25:39.948733 30761 heartbeater.cc:464] Registering TS with master...
I20260929 20:25:39.949005 30761 heartbeater.cc:511] Master 127.17.53.126:46595 requested a full tablet report, sending...
I20260929 20:25:39.950419 30761 heartbeater.cc:464] Registering TS with master...
I20260929 20:25:39.950836 30761 heartbeater.cc:511] Master 127.17.53.126:46595 requested a full tablet report, sending...
W20260929 20:25:39.952189 30761 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 20:25:40.290501 30765 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:25:40.290807 30765 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:25:40.290988 30765 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:25:40.299039 30765 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:25:40.299168 30765 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:25:40.299345 30765 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.53.66
I20260929 20:25:40.311282 30765 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.53.84:39811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713339161492-17621-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713339161492-17621-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.17.53.84:58281/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.66.keytab
--principal=kudu/127.17.53.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.53.66:43573
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713339161492-17621-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.17.53.66
--webserver_port=34809
--tserver_master_addrs=127.17.53.126:46595
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.53.66
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713339161492-17621-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:25:40.312633 30765 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:25:40.314002 30765 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:25:40.329049 30778 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:25:40.329870 30777 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:25:40.330377 30780 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:25:40.333717 30765 server_base.cc:1034] running on GCE node
I20260929 20:25:40.334362 30765 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:25:40.336377 30765 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:25:40.337677 30765 hybrid_clock.cc:648] HybridClock initialized: now 1790713540337602 us; error 57 us; skew 500 ppm
I20260929 20:25:40.338032 30765 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:25:40 dist-test-slave-zbnc krb5kdc[29548](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713540, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.53.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:25:40.342564 30765 init.cc:377] Logged in from keytab as kudu/127.17.53.66@KRBTEST.COM (short username kudu)
I20260929 20:25:40.348205 30765 webserver.cc:460] Webserver started at http://127.17.53.66:34809/ using document root <none> and password file <none>
I20260929 20:25:40.348973 30765 fs_manager.cc:359] Metadata directory not provided
I20260929 20:25:40.349140 30765 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 29 20:25:40 dist-test-slave-zbnc krb5kdc[29548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713540, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.53.66@KRBTEST.COM for HTTP/127.17.53.84@KRBTEST.COM
I20260929 20:25:40.382431 30765 fs_manager.cc:709] Time spent opening directory manager: real 0.009s user 0.004s sys 0.004s
I20260929 20:25:40.385963 30790 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:25:40.387583 30765 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 20:25:40.387784 30765 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713339161492-17621-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713339161492-17621-0/minicluster-data/ts-1/wal
uuid: "6701d4dcd11647d9801a1e2ef14e2bcd"
format_stamp: "Formatted at 2026-09-29 20:25:37 on dist-test-slave-zbnc"
server_key: "2bd3eb700de2c85ff0fb3ff5b2834c4d"
server_key_iv: "9c36d20fa41e65f417068ae62c4b4634"
server_key_version: "kuduclusterkey@0"
I20260929 20:25:40.388835 30765 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713339161492-17621-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713339161492-17621-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713339161492-17621-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:25:40.420020 30765 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:25:40.425482 30765 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:25:40.425863 30765 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:25:40.429814 30765 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:25:40.431723 30765 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:25:40.431836 30765 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:25:40.431938 30765 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:25:40.432015 30765 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:25:40.507051 30765 rpc_server.cc:307] RPC server started. Bound to: 127.17.53.66:43573
I20260929 20:25:40.507094 30904 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.53.66:43573 every 8 connection(s)
I20260929 20:25:40.508955 30765 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713339161492-17621-0/minicluster-data/ts-1/data/info.pb
I20260929 20:25:40.518743 17621 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu as pid 30765
I20260929 20:25:40.519791 17621 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
/tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713339161492-17621-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713339161492-17621-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713339161492-17621-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713339161492-17621-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.17.53.84:58281/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.53.67:46175
--local_ip_for_outbound_sockets=127.17.53.67
--tserver_master_addrs=127.17.53.126:46595
--webserver_port=43685
--webserver_interface=127.17.53.67
--keytab_file=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.67.keytab
--principal=kudu/127.17.53.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.53.84:39811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.17.53.84:58281/kms with env {KRB5CCNAME=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713339161492-17621-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713339161492-17621-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713339161492-17621-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 20:25:40 dist-test-slave-zbnc krb5kdc[29548](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790713540, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.53.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 29 20:25:40 dist-test-slave-zbnc krb5kdc[29548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713540, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.53.66@KRBTEST.COM for kudu/127.17.53.126@KRBTEST.COM
I20260929 20:25:40.549692 30907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:25:40.513485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:46595 (local address 127.17.53.66:44307)
0929 20:25:40.523501 (+ 10016us) negotiation.cc:107] Waiting for socket to connect
0929 20:25:40.523568 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0929 20:25:40.526248 (+ 2680us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:25:40.528155 (+ 1907us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:25:40.528170 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:25:40.528853 (+ 683us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:25:40.529706 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:25:40.529734 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:25:40.531402 (+ 1668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:25:40.531406 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:25:40.533033 (+ 1627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:25:40.533046 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:25:40.535190 (+ 2144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:25:40.535201 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:25:40.535547 (+ 346us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:25:40.536367 (+ 820us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:25:40.536403 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:25:40.539384 (+ 2981us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:25:40.544075 (+ 4691us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:25:40.544089 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:25:40.544115 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:25:40.544805 (+ 690us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:25:40.545288 (+ 483us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:25:40.545296 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:25:40.545303 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:25:40.545571 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:25:40.546382 (+ 811us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:25:40.546393 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:25:40.546956 (+ 563us) client_negotiation.cc:769] Sending connection context
0929 20:25:40.547313 (+ 357us) client_negotiation.cc:240] Negotiation successful
0929 20:25:40.548018 (+ 705us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9640,"thread_start_us":9545,"threads_started":1}
I20260929 20:25:40.551052 30908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:25:40.514262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.53.66:44307 (local address 127.17.53.126:46595)
0929 20:25:40.527521 (+ 13259us) server_negotiation.cc:206] Beginning negotiation
0929 20:25:40.527529 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 20:25:40.527553 (+ 24us) server_negotiation.cc:407] Connection header received
0929 20:25:40.527619 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:25:40.527625 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:25:40.527709 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:25:40.527830 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:25:40.529922 (+ 2092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:25:40.531225 (+ 1303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:25:40.533183 (+ 1958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:25:40.534947 (+ 1764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:25:40.535096 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:25:40.539700 (+ 4604us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:25:40.539710 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:25:40.539718 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:25:40.539767 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:25:40.543835 (+ 4068us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:25:40.544961 (+ 1126us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:25:40.544970 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:25:40.544977 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:25:40.545139 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:25:40.545727 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:25:40.545735 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:25:40.545741 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:25:40.546145 (+ 404us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:25:40.546293 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0929 20:25:40.550232 (+ 3939us) server_negotiation.cc:299] Negotiation successful
0929 20:25:40.550521 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12524,"thread_start_us":12413,"threads_started":1}
I20260929 20:25:40.553541 30905 heartbeater.cc:347] Connected to a master server at 127.17.53.126:46595
I20260929 20:25:40.553905 30905 heartbeater.cc:464] Registering TS with master...
I20260929 20:25:40.554937 30905 heartbeater.cc:511] Master 127.17.53.126:46595 requested a full tablet report, sending...
I20260929 20:25:40.557142 30563 ts_manager.cc:194] Registered new tserver with Master: 6701d4dcd11647d9801a1e2ef14e2bcd (127.17.53.66:43573)
I20260929 20:25:40.559396 30563 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.53.66@KRBTEST.COM'} at 127.17.53.66:44307
W20260929 20:25:40.947144 30909 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:25:40.947602 30909 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:25:40.947779 30909 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260929 20:25:40.952346 30761 heartbeater.cc:464] Registering TS with master...
I20260929 20:25:40.952795 30761 heartbeater.cc:511] Master 127.17.53.126:46595 requested a full tablet report, sending...
I20260929 20:25:40.954056 30563 ts_manager.cc:194] Registered new tserver with Master: 7b4c9e1d97384eaa8121341ae228d059 (127.17.53.65:33033)
I20260929 20:25:40.955206 30563 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.53.65@KRBTEST.COM'} at 127.17.53.65:40929
W20260929 20:25:40.959514 30909 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:25:40.959692 30909 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:25:40.959918 30909 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.53.67
I20260929 20:25:40.975570 30909 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.53.84:39811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713339161492-17621-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713339161492-17621-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.17.53.84:58281/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.67.keytab
--principal=kudu/127.17.53.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.53.67:46175
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713339161492-17621-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.17.53.67
--webserver_port=43685
--tserver_master_addrs=127.17.53.126:46595
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.53.67
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713339161492-17621-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:25:40.977450 30909 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:25:40.979233 30909 file_cache.cc:493] Constructed file cache file cache with capacity 419430
Sep 29 20:25:40 dist-test-slave-zbnc krb5kdc[29548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713540, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.53.126@KRBTEST.COM for HTTP/127.17.53.84@KRBTEST.COM
W20260929 20:25:41.012256 30916 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:25:41.014321 30909 server_base.cc:1034] running on GCE node
W20260929 20:25:41.015324 30915 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:25:41.015637 30919 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:25:41.016697 30909 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:25:41.019104 30909 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:25:41.020346 30909 hybrid_clock.cc:648] HybridClock initialized: now 1790713541020274 us; error 61 us; skew 500 ppm
I20260929 20:25:41.020773 30909 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:25:41 dist-test-slave-zbnc krb5kdc[29548](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713541, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.53.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:25:41.025202 30909 init.cc:377] Logged in from keytab as kudu/127.17.53.67@KRBTEST.COM (short username kudu)
I20260929 20:25:41.033360 30909 webserver.cc:460] Webserver started at http://127.17.53.67:43685/ using document root <none> and password file <none>
I20260929 20:25:41.034185 30909 fs_manager.cc:359] Metadata directory not provided
I20260929 20:25:41.034359 30909 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 29 20:25:41 dist-test-slave-zbnc krb5kdc[29548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713541, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.53.67@KRBTEST.COM for HTTP/127.17.53.84@KRBTEST.COM
I20260929 20:25:41.074291 30909 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.005s sys 0.001s
I20260929 20:25:41.078199 30926 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:25:41.080082 30909 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.003s
I20260929 20:25:41.080255 30909 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713339161492-17621-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713339161492-17621-0/minicluster-data/ts-2/wal
uuid: "3cd60b52e8654707a72d84c66e8a31cb"
format_stamp: "Formatted at 2026-09-29 20:25:38 on dist-test-slave-zbnc"
server_key: "c75d4942ec6ff83e70877a4afc3e7fb5"
server_key_iv: "628ebfae33a18a360c67fe960aee90f0"
server_key_version: "kuduclusterkey@0"
I20260929 20:25:41.081561 30909 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713339161492-17621-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713339161492-17621-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713339161492-17621-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:25:41.126415 30909 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:25:41.132016 30909 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:25:41.132326 30909 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:25:41.133610 30909 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:25:41.137308 30909 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:25:41.137493 30909 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260929 20:25:41.137679 30909 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:25:41.137804 30909 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:25:41.182350 30909 rpc_server.cc:307] RPC server started. Bound to: 127.17.53.67:46175
I20260929 20:25:41.182402 31038 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.53.67:46175 every 8 connection(s)
I20260929 20:25:41.184340 30909 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713339161492-17621-0/minicluster-data/ts-2/data/info.pb
I20260929 20:25:41.194279 17621 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu as pid 30909
Sep 29 20:25:41 dist-test-slave-zbnc krb5kdc[29548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713541, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.53.67@KRBTEST.COM for kudu/127.17.53.126@KRBTEST.COM
I20260929 20:25:41.225317 31044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:25:41.195778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51394 (local address 127.17.53.126:46595)
0929 20:25:41.207973 (+ 12195us) server_negotiation.cc:206] Beginning negotiation
0929 20:25:41.207980 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:25:41.207999 (+ 19us) server_negotiation.cc:407] Connection header received
0929 20:25:41.208060 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:25:41.208068 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:25:41.208153 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:25:41.208258 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:25:41.210619 (+ 2361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:25:41.211811 (+ 1192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:25:41.213690 (+ 1879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:25:41.214390 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:25:41.214487 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:25:41.217973 (+ 3486us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:25:41.217979 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:25:41.217983 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:25:41.218019 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:25:41.221531 (+ 3512us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:25:41.222380 (+ 849us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:25:41.222385 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:25:41.222388 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:25:41.222531 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:25:41.223474 (+ 943us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:25:41.223478 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:25:41.223481 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:25:41.223814 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:25:41.224080 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0929 20:25:41.224632 (+ 552us) server_negotiation.cc:299] Negotiation successful
0929 20:25:41.224944 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":26,"server-negotiator.queue_time_us":12075,"thread_start_us":12369,"threads_started":1}
I20260929 20:25:41.229560 31041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:25:41.187229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.53.67:53529 (local address 127.17.53.126:46595)
0929 20:25:41.206771 (+ 19542us) server_negotiation.cc:206] Beginning negotiation
0929 20:25:41.206778 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:25:41.206801 (+ 23us) server_negotiation.cc:407] Connection header received
0929 20:25:41.206911 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:25:41.206918 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:25:41.207028 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:25:41.207183 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:25:41.209551 (+ 2368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:25:41.210583 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:25:41.213048 (+ 2465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:25:41.213793 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:25:41.213930 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:25:41.221494 (+ 7564us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:25:41.221501 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:25:41.221506 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:25:41.221544 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:25:41.225388 (+ 3844us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:25:41.226382 (+ 994us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:25:41.226387 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:25:41.226390 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:25:41.226536 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:25:41.227110 (+ 574us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:25:41.227115 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:25:41.227119 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:25:41.227438 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:25:41.227571 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0929 20:25:41.229035 (+ 1464us) server_negotiation.cc:299] Negotiation successful
0929 20:25:41.229235 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":37,"server-negotiator.queue_time_us":19401,"thread_start_us":20992,"threads_started":1}
I20260929 20:25:41.230429 31042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:25:41.188075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:46595 (local address 127.17.53.67:53529)
0929 20:25:41.199265 (+ 11190us) negotiation.cc:107] Waiting for socket to connect
0929 20:25:41.199317 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0929 20:25:41.201480 (+ 2163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:25:41.207746 (+ 6266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:25:41.207761 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:25:41.208510 (+ 749us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:25:41.209336 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:25:41.209361 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:25:41.210771 (+ 1410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:25:41.210776 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:25:41.212851 (+ 2075us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:25:41.212881 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:25:41.214292 (+ 1411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:25:41.214300 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:25:41.214667 (+ 367us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:25:41.217536 (+ 2869us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:25:41.217574 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:25:41.221132 (+ 3558us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:25:41.225561 (+ 4429us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:25:41.225571 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:25:41.225591 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:25:41.226220 (+ 629us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:25:41.226677 (+ 457us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:25:41.226681 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:25:41.226685 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:25:41.226968 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:25:41.227643 (+ 675us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:25:41.227652 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:25:41.228121 (+ 469us) client_negotiation.cc:769] Sending connection context
0929 20:25:41.228420 (+ 299us) client_negotiation.cc:240] Negotiation successful
0929 20:25:41.228961 (+ 541us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10855,"thread_start_us":19085,"threads_started":1}
I20260929 20:25:41.232370 31039 heartbeater.cc:347] Connected to a master server at 127.17.53.126:46595
I20260929 20:25:41.232683 31039 heartbeater.cc:464] Registering TS with master...
I20260929 20:25:41.233573 31039 heartbeater.cc:511] Master 127.17.53.126:46595 requested a full tablet report, sending...
I20260929 20:25:41.235477 30563 ts_manager.cc:194] Registered new tserver with Master: 3cd60b52e8654707a72d84c66e8a31cb (127.17.53.67:46175)
I20260929 20:25:41.236565 30563 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.53.67@KRBTEST.COM'} at 127.17.53.67:53529
I20260929 20:25:41.241143 17621 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 20:25:41.261898 31041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:25:41.244366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51400 (local address 127.17.53.126:46595)
0929 20:25:41.244560 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0929 20:25:41.244569 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 20:25:41.247472 (+ 2903us) server_negotiation.cc:407] Connection header received
0929 20:25:41.247689 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:25:41.247694 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:25:41.247785 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:25:41.247889 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:25:41.249446 (+ 1557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:25:41.250155 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:25:41.251637 (+ 1482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:25:41.252079 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:25:41.252211 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:25:41.254850 (+ 2639us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:25:41.254884 (+ 34us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:25:41.254891 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:25:41.254932 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:25:41.258332 (+ 3400us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:25:41.259268 (+ 936us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:25:41.259274 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:25:41.259278 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:25:41.259431 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:25:41.259880 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:25:41.259885 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:25:41.259888 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:25:41.260262 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:25:41.260542 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0929 20:25:41.261304 (+ 762us) server_negotiation.cc:299] Negotiation successful
0929 20:25:41.261545 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260929 20:25:41.268692 30563 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51400:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 20:25:41.279273 30563 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 20:25:41.317545 31055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:25:41.300094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.67:46175 (local address 127.0.0.1:43536)
0929 20:25:41.305562 (+ 5468us) negotiation.cc:107] Waiting for socket to connect
0929 20:25:41.305619 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0929 20:25:41.305842 (+ 223us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:25:41.307149 (+ 1307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:25:41.307166 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:25:41.307205 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:25:41.307679 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:25:41.307696 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:25:41.309871 (+ 2175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:25:41.309880 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:25:41.312551 (+ 2671us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:25:41.312570 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:25:41.316295 (+ 3725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:25:41.316304 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:25:41.316592 (+ 288us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:25:41.316659 (+ 67us) client_negotiation.cc:769] Sending connection context
0929 20:25:41.316795 (+ 136us) client_negotiation.cc:240] Negotiation successful
0929 20:25:41.317052 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5155,"thread_start_us":8951,"threads_started":1}
I20260929 20:25:41.318558 31056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:25:41.300799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43536 (local address 127.17.53.67:46175)
0929 20:25:41.304916 (+ 4117us) server_negotiation.cc:206] Beginning negotiation
0929 20:25:41.304930 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 20:25:41.306094 (+ 1164us) server_negotiation.cc:407] Connection header received
0929 20:25:41.306238 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:25:41.306249 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:25:41.306558 (+ 309us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:25:41.306764 (+ 206us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:25:41.307925 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:25:41.309712 (+ 1787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:25:41.312766 (+ 3054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:25:41.315670 (+ 2904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:25:41.315820 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:25:41.316116 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0929 20:25:41.317998 (+ 1882us) server_negotiation.cc:299] Negotiation successful
0929 20:25:41.318191 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3905,"thread_start_us":6200,"threads_started":1}
I20260929 20:25:41.326355 30974 tablet_service.cc:1467] Processing CreateTablet for tablet 85973a15d53e452ebe62ea6db874dfe1 (DEFAULT_TABLE table=test-table [id=7ef64d7264e04c03b42469be462ce67e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:25:41.328441 30974 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 85973a15d53e452ebe62ea6db874dfe1. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 29 20:25:41 dist-test-slave-zbnc krb5kdc[29548](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713540, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.53.126@KRBTEST.COM for HTTP/127.17.53.84@KRBTEST.COM
I20260929 20:25:41.353242 31055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:25:41.304079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.66:43573 (local address 127.0.0.1:36126)
0929 20:25:41.319177 (+ 15098us) negotiation.cc:107] Waiting for socket to connect
0929 20:25:41.319196 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 20:25:41.319307 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:25:41.334049 (+ 14742us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:25:41.334057 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:25:41.334099 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:25:41.334422 (+ 323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:25:41.334437 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:25:41.336926 (+ 2489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:25:41.336935 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:25:41.346748 (+ 9813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:25:41.346774 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:25:41.352376 (+ 5602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:25:41.352382 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:25:41.352593 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:25:41.352641 (+ 48us) client_negotiation.cc:769] Sending connection context
0929 20:25:41.352718 (+ 77us) client_negotiation.cc:240] Negotiation successful
0929 20:25:41.352904 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14944,"thread_start_us":16289,"threads_started":1}
I20260929 20:25:41.376084 31058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:25:41.304642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36126 (local address 127.17.53.66:43573)
0929 20:25:41.322168 (+ 17526us) server_negotiation.cc:206] Beginning negotiation
0929 20:25:41.322179 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 20:25:41.322213 (+ 34us) server_negotiation.cc:407] Connection header received
0929 20:25:41.322318 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:25:41.322326 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:25:41.322587 (+ 261us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:25:41.322810 (+ 223us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:25:41.334678 (+ 11868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:25:41.336685 (+ 2007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:25:41.347245 (+ 10560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:25:41.352079 (+ 4834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:25:41.352246 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:25:41.354473 (+ 2227us) server_negotiation.cc:1035] Waiting for connection context
0929 20:25:41.354560 (+ 87us) server_negotiation.cc:299] Negotiation successful
0929 20:25:41.354730 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16790,"thread_start_us":21398,"threads_started":1}
I20260929 20:25:41.379983 31060 tablet_bootstrap.cc:492] T 85973a15d53e452ebe62ea6db874dfe1 P 3cd60b52e8654707a72d84c66e8a31cb: Bootstrap starting.
I20260929 20:25:41.380407 30840 tablet_service.cc:1467] Processing CreateTablet for tablet 85973a15d53e452ebe62ea6db874dfe1 (DEFAULT_TABLE table=test-table [id=7ef64d7264e04c03b42469be462ce67e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:25:41.381987 30840 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 85973a15d53e452ebe62ea6db874dfe1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:25:41.383996 31060 tablet_bootstrap.cc:654] T 85973a15d53e452ebe62ea6db874dfe1 P 3cd60b52e8654707a72d84c66e8a31cb: Neither blocks nor log segments found. Creating new log.
I20260929 20:25:41.385452 31060 log.cc:824] T 85973a15d53e452ebe62ea6db874dfe1 P 3cd60b52e8654707a72d84c66e8a31cb: Log is configured to *not* fsync() on all Append() calls
I20260929 20:25:41.396972 31057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:25:41.327422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.65:33033 (local address 127.0.0.1:32934)
0929 20:25:41.330713 (+ 3291us) negotiation.cc:107] Waiting for socket to connect
0929 20:25:41.330733 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 20:25:41.330880 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:25:41.354977 (+ 24097us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:25:41.354984 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:25:41.355017 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:25:41.355963 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:25:41.355976 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:25:41.377707 (+ 21731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:25:41.377713 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:25:41.379787 (+ 2074us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:25:41.379807 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:25:41.390674 (+ 10867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:25:41.390682 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:25:41.396283 (+ 5601us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:25:41.396334 (+ 51us) client_negotiation.cc:769] Sending connection context
0929 20:25:41.396458 (+ 124us) client_negotiation.cc:240] Negotiation successful
0929 20:25:41.396627 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3082}
I20260929 20:25:41.400002 31059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:25:41.322576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32934 (local address 127.17.53.65:33033)
0929 20:25:41.351467 (+ 28891us) server_negotiation.cc:206] Beginning negotiation
0929 20:25:41.351477 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 20:25:41.351697 (+ 220us) server_negotiation.cc:407] Connection header received
0929 20:25:41.351814 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:25:41.351825 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:25:41.353795 (+ 1970us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:25:41.356185 (+ 2390us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:25:41.356437 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:25:41.377538 (+ 21101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:25:41.387493 (+ 9955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:25:41.389077 (+ 1584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:25:41.389234 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:25:41.389474 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0929 20:25:41.399324 (+ 9850us) server_negotiation.cc:299] Negotiation successful
0929 20:25:41.399548 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28234,"thread_start_us":27731,"threads_started":1}
I20260929 20:25:41.402556 31060 tablet_bootstrap.cc:492] T 85973a15d53e452ebe62ea6db874dfe1 P 3cd60b52e8654707a72d84c66e8a31cb: No bootstrap required, opened a new log
I20260929 20:25:41.402911 31060 ts_tablet_manager.cc:1397] T 85973a15d53e452ebe62ea6db874dfe1 P 3cd60b52e8654707a72d84c66e8a31cb: Time spent bootstrapping tablet: real 0.023s user 0.014s sys 0.005s
I20260929 20:25:41.404613 30695 tablet_service.cc:1467] Processing CreateTablet for tablet 85973a15d53e452ebe62ea6db874dfe1 (DEFAULT_TABLE table=test-table [id=7ef64d7264e04c03b42469be462ce67e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:25:41.406132 30695 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 85973a15d53e452ebe62ea6db874dfe1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:25:41.425200 31060 raft_consensus.cc:348] T 85973a15d53e452ebe62ea6db874dfe1 P 3cd60b52e8654707a72d84c66e8a31cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cd60b52e8654707a72d84c66e8a31cb" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 46175 } } peers { permanent_uuid: "6701d4dcd11647d9801a1e2ef14e2bcd" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 43573 } } peers { permanent_uuid: "7b4c9e1d97384eaa8121341ae228d059" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 33033 } }
I20260929 20:25:41.425735 31060 raft_consensus.cc:374] T 85973a15d53e452ebe62ea6db874dfe1 P 3cd60b52e8654707a72d84c66e8a31cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:25:41.425832 31060 raft_consensus.cc:729] T 85973a15d53e452ebe62ea6db874dfe1 P 3cd60b52e8654707a72d84c66e8a31cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3cd60b52e8654707a72d84c66e8a31cb, State: Initialized, Role: FOLLOWER
I20260929 20:25:41.426545 31060 consensus_queue.cc:260] T 85973a15d53e452ebe62ea6db874dfe1 P 3cd60b52e8654707a72d84c66e8a31cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cd60b52e8654707a72d84c66e8a31cb" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 46175 } } peers { permanent_uuid: "6701d4dcd11647d9801a1e2ef14e2bcd" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 43573 } } peers { permanent_uuid: "7b4c9e1d97384eaa8121341ae228d059" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 33033 } }
I20260929 20:25:41.439760 31062 tablet_bootstrap.cc:492] T 85973a15d53e452ebe62ea6db874dfe1 P 6701d4dcd11647d9801a1e2ef14e2bcd: Bootstrap starting.
I20260929 20:25:41.444967 31062 tablet_bootstrap.cc:654] T 85973a15d53e452ebe62ea6db874dfe1 P 6701d4dcd11647d9801a1e2ef14e2bcd: Neither blocks nor log segments found. Creating new log.
I20260929 20:25:41.446457 31062 log.cc:824] T 85973a15d53e452ebe62ea6db874dfe1 P 6701d4dcd11647d9801a1e2ef14e2bcd: Log is configured to *not* fsync() on all Append() calls
I20260929 20:25:41.447578 31039 heartbeater.cc:503] Master 127.17.53.126:46595 was elected leader, sending a full tablet report...
I20260929 20:25:41.448678 31060 ts_tablet_manager.cc:1428] T 85973a15d53e452ebe62ea6db874dfe1 P 3cd60b52e8654707a72d84c66e8a31cb: Time spent starting tablet: real 0.046s user 0.009s sys 0.006s
I20260929 20:25:41.464725 31064 tablet_bootstrap.cc:492] T 85973a15d53e452ebe62ea6db874dfe1 P 7b4c9e1d97384eaa8121341ae228d059: Bootstrap starting.
I20260929 20:25:41.466547 31062 tablet_bootstrap.cc:492] T 85973a15d53e452ebe62ea6db874dfe1 P 6701d4dcd11647d9801a1e2ef14e2bcd: No bootstrap required, opened a new log
I20260929 20:25:41.466966 31062 ts_tablet_manager.cc:1397] T 85973a15d53e452ebe62ea6db874dfe1 P 6701d4dcd11647d9801a1e2ef14e2bcd: Time spent bootstrapping tablet: real 0.027s user 0.006s sys 0.008s
I20260929 20:25:41.469846 31064 tablet_bootstrap.cc:654] T 85973a15d53e452ebe62ea6db874dfe1 P 7b4c9e1d97384eaa8121341ae228d059: Neither blocks nor log segments found. Creating new log.
I20260929 20:25:41.477653 31062 raft_consensus.cc:348] T 85973a15d53e452ebe62ea6db874dfe1 P 6701d4dcd11647d9801a1e2ef14e2bcd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cd60b52e8654707a72d84c66e8a31cb" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 46175 } } peers { permanent_uuid: "6701d4dcd11647d9801a1e2ef14e2bcd" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 43573 } } peers { permanent_uuid: "7b4c9e1d97384eaa8121341ae228d059" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 33033 } }
I20260929 20:25:41.479815 31062 raft_consensus.cc:374] T 85973a15d53e452ebe62ea6db874dfe1 P 6701d4dcd11647d9801a1e2ef14e2bcd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:25:41.479993 31062 raft_consensus.cc:729] T 85973a15d53e452ebe62ea6db874dfe1 P 6701d4dcd11647d9801a1e2ef14e2bcd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6701d4dcd11647d9801a1e2ef14e2bcd, State: Initialized, Role: FOLLOWER
I20260929 20:25:41.480799 31062 consensus_queue.cc:260] T 85973a15d53e452ebe62ea6db874dfe1 P 6701d4dcd11647d9801a1e2ef14e2bcd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cd60b52e8654707a72d84c66e8a31cb" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 46175 } } peers { permanent_uuid: "6701d4dcd11647d9801a1e2ef14e2bcd" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 43573 } } peers { permanent_uuid: "7b4c9e1d97384eaa8121341ae228d059" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 33033 } }
I20260929 20:25:41.484699 31064 log.cc:824] T 85973a15d53e452ebe62ea6db874dfe1 P 7b4c9e1d97384eaa8121341ae228d059: Log is configured to *not* fsync() on all Append() calls
I20260929 20:25:41.493410 31064 tablet_bootstrap.cc:492] T 85973a15d53e452ebe62ea6db874dfe1 P 7b4c9e1d97384eaa8121341ae228d059: No bootstrap required, opened a new log
I20260929 20:25:41.493839 31064 ts_tablet_manager.cc:1397] T 85973a15d53e452ebe62ea6db874dfe1 P 7b4c9e1d97384eaa8121341ae228d059: Time spent bootstrapping tablet: real 0.029s user 0.011s sys 0.000s
I20260929 20:25:41.519672 30905 heartbeater.cc:503] Master 127.17.53.126:46595 was elected leader, sending a full tablet report...
I20260929 20:25:41.520438 31062 ts_tablet_manager.cc:1428] T 85973a15d53e452ebe62ea6db874dfe1 P 6701d4dcd11647d9801a1e2ef14e2bcd: Time spent starting tablet: real 0.053s user 0.026s sys 0.008s
I20260929 20:25:41.520280 31064 raft_consensus.cc:348] T 85973a15d53e452ebe62ea6db874dfe1 P 7b4c9e1d97384eaa8121341ae228d059 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cd60b52e8654707a72d84c66e8a31cb" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 46175 } } peers { permanent_uuid: "6701d4dcd11647d9801a1e2ef14e2bcd" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 43573 } } peers { permanent_uuid: "7b4c9e1d97384eaa8121341ae228d059" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 33033 } }
I20260929 20:25:41.520924 31064 raft_consensus.cc:374] T 85973a15d53e452ebe62ea6db874dfe1 P 7b4c9e1d97384eaa8121341ae228d059 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:25:41.521122 31064 raft_consensus.cc:729] T 85973a15d53e452ebe62ea6db874dfe1 P 7b4c9e1d97384eaa8121341ae228d059 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b4c9e1d97384eaa8121341ae228d059, State: Initialized, Role: FOLLOWER
I20260929 20:25:41.521907 31064 consensus_queue.cc:260] T 85973a15d53e452ebe62ea6db874dfe1 P 7b4c9e1d97384eaa8121341ae228d059 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cd60b52e8654707a72d84c66e8a31cb" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 46175 } } peers { permanent_uuid: "6701d4dcd11647d9801a1e2ef14e2bcd" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 43573 } } peers { permanent_uuid: "7b4c9e1d97384eaa8121341ae228d059" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 33033 } }
I20260929 20:25:41.526798 30761 heartbeater.cc:503] Master 127.17.53.126:46595 was elected leader, sending a full tablet report...
I20260929 20:25:41.527890 31064 ts_tablet_manager.cc:1428] T 85973a15d53e452ebe62ea6db874dfe1 P 7b4c9e1d97384eaa8121341ae228d059: Time spent starting tablet: real 0.034s user 0.014s sys 0.000s
W20260929 20:25:41.528633 30906 tablet.cc:2369] T 85973a15d53e452ebe62ea6db874dfe1 P 6701d4dcd11647d9801a1e2ef14e2bcd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 20:25:41.573215 31066 raft_consensus.cc:482] T 85973a15d53e452ebe62ea6db874dfe1 P 6701d4dcd11647d9801a1e2ef14e2bcd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 20:25:41.573602 31066 raft_consensus.cc:504] T 85973a15d53e452ebe62ea6db874dfe1 P 6701d4dcd11647d9801a1e2ef14e2bcd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cd60b52e8654707a72d84c66e8a31cb" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 46175 } } peers { permanent_uuid: "6701d4dcd11647d9801a1e2ef14e2bcd" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 43573 } } peers { permanent_uuid: "7b4c9e1d97384eaa8121341ae228d059" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 33033 } }
I20260929 20:25:41.575665 31066 leader_election.cc:288] T 85973a15d53e452ebe62ea6db874dfe1 P 6701d4dcd11647d9801a1e2ef14e2bcd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3cd60b52e8654707a72d84c66e8a31cb (127.17.53.67:46175), 7b4c9e1d97384eaa8121341ae228d059 (127.17.53.65:33033)
I20260929 20:25:41.597754 31056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:25:41.576551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.53.66:42257 (local address 127.17.53.67:46175)
0929 20:25:41.576782 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0929 20:25:41.576792 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 20:25:41.588509 (+ 11717us) server_negotiation.cc:407] Connection header received
0929 20:25:41.588857 (+ 348us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:25:41.588866 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:25:41.588948 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:25:41.589028 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:25:41.589894 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:25:41.591031 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:25:41.595081 (+ 4050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:25:41.596299 (+ 1218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:25:41.596461 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:25:41.596541 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0929 20:25:41.597280 (+ 739us) server_negotiation.cc:299] Negotiation successful
0929 20:25:41.597424 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 20:25:41.598719 31069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:25:41.576037 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.67:46175 (local address 127.17.53.66:42257)
0929 20:25:41.588359 (+ 12322us) negotiation.cc:107] Waiting for socket to connect
0929 20:25:41.588389 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0929 20:25:41.588654 (+ 265us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:25:41.589243 (+ 589us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:25:41.589247 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:25:41.589272 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:25:41.589688 (+ 416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:25:41.589701 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:25:41.591231 (+ 1530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:25:41.591239 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:25:41.594887 (+ 3648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:25:41.594905 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:25:41.596798 (+ 1893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:25:41.596805 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:25:41.597063 (+ 258us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:25:41.597101 (+ 38us) client_negotiation.cc:769] Sending connection context
0929 20:25:41.598113 (+ 1012us) client_negotiation.cc:240] Negotiation successful
0929 20:25:41.598271 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11412,"mutex_wait_us":4,"spinlock_wait_cycles":704896,"thread_start_us":11490,"threads_started":1}
I20260929 20:25:41.599819 30994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85973a15d53e452ebe62ea6db874dfe1" candidate_uuid: "6701d4dcd11647d9801a1e2ef14e2bcd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3cd60b52e8654707a72d84c66e8a31cb" is_pre_election: true
I20260929 20:25:41.600492 30994 raft_consensus.cc:2454] T 85973a15d53e452ebe62ea6db874dfe1 P 3cd60b52e8654707a72d84c66e8a31cb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6701d4dcd11647d9801a1e2ef14e2bcd in term 0.
I20260929 20:25:41.601217 31059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:25:41.576636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.53.66:60407 (local address 127.17.53.65:33033)
0929 20:25:41.576887 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0929 20:25:41.576894 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:25:41.588648 (+ 11754us) server_negotiation.cc:407] Connection header received
0929 20:25:41.589104 (+ 456us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:25:41.589109 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:25:41.589192 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:25:41.589256 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:25:41.590076 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:25:41.591137 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:25:41.598450 (+ 7313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:25:41.599848 (+ 1398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:25:41.600010 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:25:41.600094 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0929 20:25:41.600790 (+ 696us) server_negotiation.cc:299] Negotiation successful
0929 20:25:41.600951 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260929 20:25:41.601392 30792 leader_election.cc:302] T 85973a15d53e452ebe62ea6db874dfe1 P 6701d4dcd11647d9801a1e2ef14e2bcd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3cd60b52e8654707a72d84c66e8a31cb, 6701d4dcd11647d9801a1e2ef14e2bcd; no voters:
I20260929 20:25:41.601939 31066 raft_consensus.cc:2781] T 85973a15d53e452ebe62ea6db874dfe1 P 6701d4dcd11647d9801a1e2ef14e2bcd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 20:25:41.602116 31066 raft_consensus.cc:482] T 85973a15d53e452ebe62ea6db874dfe1 P 6701d4dcd11647d9801a1e2ef14e2bcd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 20:25:41.602246 31066 raft_consensus.cc:3037] T 85973a15d53e452ebe62ea6db874dfe1 P 6701d4dcd11647d9801a1e2ef14e2bcd [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:25:41.604516 31070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:25:41.576208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.65:33033 (local address 127.17.53.66:60407)
0929 20:25:41.588542 (+ 12334us) negotiation.cc:107] Waiting for socket to connect
0929 20:25:41.588559 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 20:25:41.588926 (+ 367us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:25:41.589477 (+ 551us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:25:41.589481 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:25:41.589502 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:25:41.589901 (+ 399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:25:41.589913 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:25:41.591312 (+ 1399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:25:41.591316 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:25:41.594941 (+ 3625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:25:41.594965 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:25:41.600313 (+ 5348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:25:41.600320 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:25:41.600506 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:25:41.600594 (+ 88us) client_negotiation.cc:769] Sending connection context
0929 20:25:41.601482 (+ 888us) client_negotiation.cc:240] Negotiation successful
0929 20:25:41.601647 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12234,"mutex_wait_us":28,"thread_start_us":12455,"threads_started":1}
I20260929 20:25:41.605719 30715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85973a15d53e452ebe62ea6db874dfe1" candidate_uuid: "6701d4dcd11647d9801a1e2ef14e2bcd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b4c9e1d97384eaa8121341ae228d059" is_pre_election: true
I20260929 20:25:41.605942 31066 raft_consensus.cc:504] T 85973a15d53e452ebe62ea6db874dfe1 P 6701d4dcd11647d9801a1e2ef14e2bcd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cd60b52e8654707a72d84c66e8a31cb" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 46175 } } peers { permanent_uuid: "6701d4dcd11647d9801a1e2ef14e2bcd" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 43573 } } peers { permanent_uuid: "7b4c9e1d97384eaa8121341ae228d059" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 33033 } }
I20260929 20:25:41.606334 30715 raft_consensus.cc:2454] T 85973a15d53e452ebe62ea6db874dfe1 P 7b4c9e1d97384eaa8121341ae228d059 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6701d4dcd11647d9801a1e2ef14e2bcd in term 0.
I20260929 20:25:41.607123 30994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85973a15d53e452ebe62ea6db874dfe1" candidate_uuid: "6701d4dcd11647d9801a1e2ef14e2bcd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3cd60b52e8654707a72d84c66e8a31cb"
I20260929 20:25:41.607367 30994 raft_consensus.cc:3037] T 85973a15d53e452ebe62ea6db874dfe1 P 3cd60b52e8654707a72d84c66e8a31cb [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:25:41.607875 30715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85973a15d53e452ebe62ea6db874dfe1" candidate_uuid: "6701d4dcd11647d9801a1e2ef14e2bcd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b4c9e1d97384eaa8121341ae228d059"
I20260929 20:25:41.608083 30715 raft_consensus.cc:3037] T 85973a15d53e452ebe62ea6db874dfe1 P 7b4c9e1d97384eaa8121341ae228d059 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:25:41.610827 30994 raft_consensus.cc:2454] T 85973a15d53e452ebe62ea6db874dfe1 P 3cd60b52e8654707a72d84c66e8a31cb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6701d4dcd11647d9801a1e2ef14e2bcd in term 1.
I20260929 20:25:41.611092 30715 raft_consensus.cc:2454] T 85973a15d53e452ebe62ea6db874dfe1 P 7b4c9e1d97384eaa8121341ae228d059 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6701d4dcd11647d9801a1e2ef14e2bcd in term 1.
I20260929 20:25:41.611339 30792 leader_election.cc:302] T 85973a15d53e452ebe62ea6db874dfe1 P 6701d4dcd11647d9801a1e2ef14e2bcd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3cd60b52e8654707a72d84c66e8a31cb, 6701d4dcd11647d9801a1e2ef14e2bcd; no voters:
I20260929 20:25:41.611572 31066 leader_election.cc:288] T 85973a15d53e452ebe62ea6db874dfe1 P 6701d4dcd11647d9801a1e2ef14e2bcd [CANDIDATE]: Term 1 election: Requested vote from peers 3cd60b52e8654707a72d84c66e8a31cb (127.17.53.67:46175), 7b4c9e1d97384eaa8121341ae228d059 (127.17.53.65:33033)
I20260929 20:25:41.611790 31066 raft_consensus.cc:2781] T 85973a15d53e452ebe62ea6db874dfe1 P 6701d4dcd11647d9801a1e2ef14e2bcd [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:25:41.611929 31066 raft_consensus.cc:686] T 85973a15d53e452ebe62ea6db874dfe1 P 6701d4dcd11647d9801a1e2ef14e2bcd [term 1 LEADER]: Becoming Leader. State: Replica: 6701d4dcd11647d9801a1e2ef14e2bcd, State: Running, Role: LEADER
I20260929 20:25:41.612439 31066 consensus_queue.cc:237] T 85973a15d53e452ebe62ea6db874dfe1 P 6701d4dcd11647d9801a1e2ef14e2bcd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cd60b52e8654707a72d84c66e8a31cb" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 46175 } } peers { permanent_uuid: "6701d4dcd11647d9801a1e2ef14e2bcd" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 43573 } } peers { permanent_uuid: "7b4c9e1d97384eaa8121341ae228d059" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 33033 } }
I20260929 20:25:41.623529 30563 catalog_manager.cc:5495] T 85973a15d53e452ebe62ea6db874dfe1 P 6701d4dcd11647d9801a1e2ef14e2bcd reported cstate change: term changed from 0 to 1, leader changed from <none> to 6701d4dcd11647d9801a1e2ef14e2bcd (127.17.53.66). New cstate: current_term: 1 leader_uuid: "6701d4dcd11647d9801a1e2ef14e2bcd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3cd60b52e8654707a72d84c66e8a31cb" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 46175 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6701d4dcd11647d9801a1e2ef14e2bcd" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 43573 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7b4c9e1d97384eaa8121341ae228d059" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 33033 } health_report { overall_health: UNKNOWN } } }
W20260929 20:25:41.628327 30762 tablet.cc:2369] T 85973a15d53e452ebe62ea6db874dfe1 P 7b4c9e1d97384eaa8121341ae228d059: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 20:25:41.687485 31040 tablet.cc:2369] T 85973a15d53e452ebe62ea6db874dfe1 P 3cd60b52e8654707a72d84c66e8a31cb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 20:25:41.691890 31058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:25:41.684860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36136 (local address 127.17.53.66:43573)
0929 20:25:41.685078 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0929 20:25:41.685084 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:25:41.685244 (+ 160us) server_negotiation.cc:407] Connection header received
0929 20:25:41.685504 (+ 260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:25:41.685508 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:25:41.685619 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:25:41.685804 (+ 185us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 20:25:41.686497 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:25:41.687548 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:25:41.689271 (+ 1723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:25:41.689892 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:25:41.690018 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:25:41.690559 (+ 541us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 20:25:41.691270 (+ 711us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 20:25:41.691418 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0929 20:25:41.691516 (+ 98us) server_negotiation.cc:299] Negotiation successful
0929 20:25:41.691599 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260929 20:25:41.701149 30994 raft_consensus.cc:1261] T 85973a15d53e452ebe62ea6db874dfe1 P 3cd60b52e8654707a72d84c66e8a31cb [term 1 FOLLOWER]: Refusing update from remote peer 6701d4dcd11647d9801a1e2ef14e2bcd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:25:41.701260 30715 raft_consensus.cc:1261] T 85973a15d53e452ebe62ea6db874dfe1 P 7b4c9e1d97384eaa8121341ae228d059 [term 1 FOLLOWER]: Refusing update from remote peer 6701d4dcd11647d9801a1e2ef14e2bcd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:25:41.702234 31071 consensus_queue.cc:1035] T 85973a15d53e452ebe62ea6db874dfe1 P 6701d4dcd11647d9801a1e2ef14e2bcd [LEADER]: Connected to new peer: Peer: permanent_uuid: "3cd60b52e8654707a72d84c66e8a31cb" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 46175 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:25:41.702950 31066 consensus_queue.cc:1035] T 85973a15d53e452ebe62ea6db874dfe1 P 6701d4dcd11647d9801a1e2ef14e2bcd [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b4c9e1d97384eaa8121341ae228d059" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 33033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:25:41.725286 31083 mvcc.cc:205] Tried to move back new op lower bound from 7334762666797948928 to 7334762666449199104. Current Snapshot: MvccSnapshot[applied={T|T < 7334762666797948928}]
I20260929 20:25:41.753557 30562 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51400:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 20:25:41.753880 30562 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51400:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 20:25:41.758152 30562 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0f40088f621e4863961f6a9da4daee82: Sending DeleteTablet for 3 replicas of tablet 85973a15d53e452ebe62ea6db874dfe1
I20260929 20:25:41.759325 30840 tablet_service.cc:1514] Processing DeleteTablet for tablet 85973a15d53e452ebe62ea6db874dfe1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:25:41 UTC) from {username='kudu', principal='kudu/127.17.53.126@KRBTEST.COM'} at 127.0.0.1:36126
I20260929 20:25:41.759419 30974 tablet_service.cc:1514] Processing DeleteTablet for tablet 85973a15d53e452ebe62ea6db874dfe1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:25:41 UTC) from {username='kudu', principal='kudu/127.17.53.126@KRBTEST.COM'} at 127.0.0.1:43536
I20260929 20:25:41.760751 30695 tablet_service.cc:1514] Processing DeleteTablet for tablet 85973a15d53e452ebe62ea6db874dfe1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:25:41 UTC) from {username='kudu', principal='kudu/127.17.53.126@KRBTEST.COM'} at 127.0.0.1:32934
I20260929 20:25:41.770231 17621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu pid 30600
I20260929 20:25:41.791121 31092 tablet_replica.cc:330] T 85973a15d53e452ebe62ea6db874dfe1 P 7b4c9e1d97384eaa8121341ae228d059: stopping tablet replica
I20260929 20:25:41.791642 31092 raft_consensus.cc:2229] T 85973a15d53e452ebe62ea6db874dfe1 P 7b4c9e1d97384eaa8121341ae228d059 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 20:25:41.792220 31092 raft_consensus.cc:2258] T 85973a15d53e452ebe62ea6db874dfe1 P 7b4c9e1d97384eaa8121341ae228d059 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:25:41.793514 31091 tablet_replica.cc:330] T 85973a15d53e452ebe62ea6db874dfe1 P 6701d4dcd11647d9801a1e2ef14e2bcd: stopping tablet replica
I20260929 20:25:41.794010 31091 raft_consensus.cc:2229] T 85973a15d53e452ebe62ea6db874dfe1 P 6701d4dcd11647d9801a1e2ef14e2bcd [term 1 LEADER]: Raft consensus shutting down.
I20260929 20:25:41.794663 31092 ts_tablet_manager.cc:1905] T 85973a15d53e452ebe62ea6db874dfe1 P 7b4c9e1d97384eaa8121341ae228d059: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:25:41.794869 31091 raft_consensus.cc:2258] T 85973a15d53e452ebe62ea6db874dfe1 P 6701d4dcd11647d9801a1e2ef14e2bcd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:25:41.797696 31091 ts_tablet_manager.cc:1905] T 85973a15d53e452ebe62ea6db874dfe1 P 6701d4dcd11647d9801a1e2ef14e2bcd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:25:41.798024 31090 tablet_replica.cc:330] T 85973a15d53e452ebe62ea6db874dfe1 P 3cd60b52e8654707a72d84c66e8a31cb: stopping tablet replica
I20260929 20:25:41.805780 31092 ts_tablet_manager.cc:1918] T 85973a15d53e452ebe62ea6db874dfe1 P 7b4c9e1d97384eaa8121341ae228d059: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 20:25:41.805799 31091 ts_tablet_manager.cc:1918] T 85973a15d53e452ebe62ea6db874dfe1 P 6701d4dcd11647d9801a1e2ef14e2bcd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 20:25:41.805967 31092 log.cc:1196] T 85973a15d53e452ebe62ea6db874dfe1 P 7b4c9e1d97384eaa8121341ae228d059: Deleting WAL directory at /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713339161492-17621-0/minicluster-data/ts-0/wal/wals/85973a15d53e452ebe62ea6db874dfe1
I20260929 20:25:41.805974 31091 log.cc:1196] T 85973a15d53e452ebe62ea6db874dfe1 P 6701d4dcd11647d9801a1e2ef14e2bcd: Deleting WAL directory at /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713339161492-17621-0/minicluster-data/ts-1/wal/wals/85973a15d53e452ebe62ea6db874dfe1
I20260929 20:25:41.806523 31092 ts_tablet_manager.cc:1939] T 85973a15d53e452ebe62ea6db874dfe1 P 7b4c9e1d97384eaa8121341ae228d059: Deleting consensus metadata
I20260929 20:25:41.806533 31091 ts_tablet_manager.cc:1939] T 85973a15d53e452ebe62ea6db874dfe1 P 6701d4dcd11647d9801a1e2ef14e2bcd: Deleting consensus metadata
I20260929 20:25:41.808583 30547 catalog_manager.cc:4841] TS 6701d4dcd11647d9801a1e2ef14e2bcd (127.17.53.66:43573): tablet 85973a15d53e452ebe62ea6db874dfe1 (table test-table [id=7ef64d7264e04c03b42469be462ce67e]) successfully deleted
I20260929 20:25:41.798419 31090 raft_consensus.cc:2229] T 85973a15d53e452ebe62ea6db874dfe1 P 3cd60b52e8654707a72d84c66e8a31cb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 20:25:41.809478 31090 raft_consensus.cc:2258] T 85973a15d53e452ebe62ea6db874dfe1 P 3cd60b52e8654707a72d84c66e8a31cb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:25:41.813009 31090 ts_tablet_manager.cc:1905] T 85973a15d53e452ebe62ea6db874dfe1 P 3cd60b52e8654707a72d84c66e8a31cb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:25:41.817132 31059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:25:41.771631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32944 (local address 127.17.53.65:33033)
0929 20:25:41.771833 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0929 20:25:41.771839 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:25:41.787596 (+ 15757us) server_negotiation.cc:407] Connection header received
0929 20:25:41.788173 (+ 577us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:25:41.788184 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:25:41.788306 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:25:41.788391 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:25:41.790847 (+ 2456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:25:41.792010 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:25:41.798543 (+ 6533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:25:41.799211 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:25:41.799324 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:25:41.809800 (+ 10476us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:25:41.809814 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:25:41.809832 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:25:41.809889 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:25:41.813592 (+ 3703us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:25:41.814507 (+ 915us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:25:41.814515 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:25:41.814519 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:25:41.814675 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:25:41.815167 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:25:41.815172 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:25:41.815176 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:25:41.815669 (+ 493us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:25:41.815945 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0929 20:25:41.816536 (+ 591us) server_negotiation.cc:299] Negotiation successful
0929 20:25:41.816772 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260929 20:25:41.818639 30756 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:25:41.821049 31090 ts_tablet_manager.cc:1918] T 85973a15d53e452ebe62ea6db874dfe1 P 3cd60b52e8654707a72d84c66e8a31cb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 20:25:41.821255 31090 log.cc:1196] T 85973a15d53e452ebe62ea6db874dfe1 P 3cd60b52e8654707a72d84c66e8a31cb: Deleting WAL directory at /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790713339161492-17621-0/minicluster-data/ts-2/wal/wals/85973a15d53e452ebe62ea6db874dfe1
I20260929 20:25:41.822007 31090 ts_tablet_manager.cc:1939] T 85973a15d53e452ebe62ea6db874dfe1 P 3cd60b52e8654707a72d84c66e8a31cb: Deleting consensus metadata
I20260929 20:25:41.824564 30548 catalog_manager.cc:4841] TS 3cd60b52e8654707a72d84c66e8a31cb (127.17.53.67:46175): tablet 85973a15d53e452ebe62ea6db874dfe1 (table test-table [id=7ef64d7264e04c03b42469be462ce67e]) successfully deleted
I20260929 20:25:41.917860 17621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu with pid 30600
I20260929 20:25:41.936237 17621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu pid 30765
W20260929 20:25:41.936239 30547 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.17.53.65:33033 (error 108)
I20260929 20:25:41.947960 31096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:25:41.942549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.65:33033 (local address 127.0.0.1:32952)
0929 20:25:41.947629 (+ 5080us) negotiation.cc:107] Waiting for socket to connect
0929 20:25:41.947771 (+ 142us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.53.65:33033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":4927,"thread_start_us":5004,"threads_started":1}
W20260929 20:25:41.948335 30547 catalog_manager.cc:4584] TS 7b4c9e1d97384eaa8121341ae228d059 (127.17.53.65:33033): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 85973a15d53e452ebe62ea6db874dfe1: Network error: Client connection negotiation failed: client connection to 127.17.53.65:33033: connect: Connection refused (error 111)
I20260929 20:25:41.961295 31058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:25:41.937113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36138 (local address 127.17.53.66:43573)
0929 20:25:41.937837 (+ 724us) server_negotiation.cc:206] Beginning negotiation
0929 20:25:41.937847 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 20:25:41.937870 (+ 23us) server_negotiation.cc:407] Connection header received
0929 20:25:41.937948 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:25:41.937957 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:25:41.938071 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:25:41.938191 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:25:41.940938 (+ 2747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:25:41.942353 (+ 1415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:25:41.945152 (+ 2799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:25:41.946064 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:25:41.946217 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:25:41.954121 (+ 7904us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:25:41.954141 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:25:41.954162 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:25:41.954223 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:25:41.958177 (+ 3954us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:25:41.959027 (+ 850us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:25:41.959041 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:25:41.959045 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:25:41.959214 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:25:41.959612 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:25:41.959619 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:25:41.959624 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:25:41.960074 (+ 450us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:25:41.960408 (+ 334us) server_negotiation.cc:1035] Waiting for connection context
0929 20:25:41.960658 (+ 250us) server_negotiation.cc:299] Negotiation successful
0929 20:25:41.960899 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":556}
I20260929 20:25:41.962065 30900 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:25:42.001920 31096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:25:42.001494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.65:33033 (local address 127.0.0.1:32962)
0929 20:25:42.001691 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0929 20:25:42.001782 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.53.65:33033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260929 20:25:42.002986 31096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:25:42.002558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.65:33033 (local address 127.0.0.1:32964)
0929 20:25:42.002702 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0929 20:25:42.002778 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.53.65:33033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260929 20:25:42.063584 31096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:25:42.063121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.65:33033 (local address 127.0.0.1:32974)
0929 20:25:42.063351 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0929 20:25:42.063438 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.53.65:33033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260929 20:25:42.064587 31096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:25:42.064242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.65:33033 (local address 127.0.0.1:32986)
0929 20:25:42.064392 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0929 20:25:42.064458 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.53.65:33033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260929 20:25:42.075814 17621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu with pid 30765
I20260929 20:25:42.099956 17621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu pid 30909
I20260929 20:25:42.117085 31098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:25:42.100785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43542 (local address 127.17.53.67:46175)
0929 20:25:42.101283 (+ 498us) server_negotiation.cc:206] Beginning negotiation
0929 20:25:42.101288 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:25:42.101304 (+ 16us) server_negotiation.cc:407] Connection header received
0929 20:25:42.101360 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:25:42.101364 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:25:42.101425 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:25:42.101529 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:25:42.103714 (+ 2185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:25:42.104802 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:25:42.106777 (+ 1975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:25:42.107437 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:25:42.107629 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:25:42.111102 (+ 3473us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:25:42.111122 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:25:42.111143 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:25:42.111202 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:25:42.114066 (+ 2864us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:25:42.115021 (+ 955us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:25:42.115029 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:25:42.115031 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:25:42.115161 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:25:42.115626 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:25:42.115629 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:25:42.115631 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:25:42.115965 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:25:42.116183 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0929 20:25:42.116615 (+ 432us) server_negotiation.cc:299] Negotiation successful
0929 20:25:42.116779 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":384,"thread_start_us":268,"threads_started":1}
I20260929 20:25:42.117846 31034 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:25:42.145468 31096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:25:42.144930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.65:33033 (local address 127.0.0.1:32998)
0929 20:25:42.145170 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0929 20:25:42.145284 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.53.65:33033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260929 20:25:42.146649 31096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:25:42.146147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.65:33033 (local address 127.0.0.1:33014)
0929 20:25:42.146388 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0929 20:25:42.146485 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.53.65:33033: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":150}
I20260929 20:25:42.208753 17621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu with pid 30909
I20260929 20:25:42.225297 17621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu pid 30530
I20260929 20:25:42.226970 30593 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:25:42.342433 17621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu with pid 30530
I20260929 20:25:42.366727 17621 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260929 20:25:42.712468 17621 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260929 20:25:42.712656 17621 mini_ranger.cc:66] Stopping Ranger...
I20260929 20:25:43.058130 17621 mini_ranger.cc:68] Stopped Ranger
2026-09-29 20:25:43.058 UTC [29578] LOG: received smart shutdown request
2026-09-29 20:25:43.061 UTC [29578] LOG: background worker "logical replication launcher" (PID 29587) exited with exit code 1
2026-09-29 20:25:43.062 UTC [29582] LOG: shutting down
2026-09-29 20:25:43.252 UTC [29578] LOG: database system is shut down
2026-09-29T20:25:43Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (53508 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790713339161492-17621-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 20:25:43 dist-test-slave-zbnc krb5kdc[31109](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 20:25:43 dist-test-slave-zbnc krb5kdc[31109](info): set up 2 sockets
Sep 29 20:25:43 dist-test-slave-zbnc krb5kdc[31109](info): commencing operation
krb5kdc: starting...
W20260929 20:25:44.488871 17621 mini_kdc.cc:121] Time spent starting KDC: real 1.066s user 0.001s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790713339161492-17621-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790713339161492-17621-0/krb5kdc/test-user.keytab.
Sep 29 20:25:44 dist-test-slave-zbnc krb5kdc[31109](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713544, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T20:25:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T20:25:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.17.53.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.53.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.53.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.126.keytab.
Entry for principal kudu/127.17.53.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.126.keytab.
I20260929 20:25:44.719291 17621 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
/tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790713339161492-17621-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790713339161492-17621-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790713339161492-17621-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790713339161492-17621-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.53.126:44735
--webserver_interface=127.17.53.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.126.keytab
--principal=kudu/127.17.53.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.53.84:36839
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.53.126:44735,127.17.53.125:37523,127.17.53.124:44835
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790713339161492-17621-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790713339161492-17621-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790713339161492-17621-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:25:45.048734 31122 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:25:45.049018 31122 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:25:45.049171 31122 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:25:45.060905 31122 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:25:45.061020 31122 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:25:45.061094 31122 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:25:45.061177 31122 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:25:45.061264 31122 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:25:45.071419 31122 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.53.84:36839
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790713339161492-17621-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790713339161492-17621-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.53.126:44735,127.17.53.125:37523,127.17.53.124:44835
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.126.keytab
--principal=kudu/127.17.53.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.53.126:44735
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790713339161492-17621-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.53.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790713339161492-17621-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:25:45.072804 31122 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:25:45.074141 31122 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:25:45.086532 31127 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:25:45.086532 31128 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:25:45.086699 31130 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:25:45.087646 31122 server_base.cc:1034] running on GCE node
I20260929 20:25:45.088475 31122 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:25:45.089993 31122 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:25:45.091214 31122 hybrid_clock.cc:648] HybridClock initialized: now 1790713545091154 us; error 53 us; skew 500 ppm
I20260929 20:25:45.091604 31122 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:25:45 dist-test-slave-zbnc krb5kdc[31109](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713545, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.53.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:25:45.094980 31122 init.cc:377] Logged in from keytab as kudu/127.17.53.126@KRBTEST.COM (short username kudu)
I20260929 20:25:45.096647 31122 webserver.cc:460] Webserver started at http://127.17.53.126:36201/ using document root <none> and password file <none>
I20260929 20:25:45.097395 31122 fs_manager.cc:359] Metadata directory not provided
I20260929 20:25:45.097548 31122 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:25:45.097908 31122 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:25:45.101078 31122 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790713339161492-17621-0/minicluster-data/master-0/data/instance:
uuid: "5ddf8d4c8aee4b1ea7bcdc4c4f712f8f"
format_stamp: "Formatted at 2026-09-29 20:25:45 on dist-test-slave-zbnc"
server_key: "ff7cd6c328f747b4e2ebbe2d6a908137"
server_key_iv: "0884ec8dd99b3a05b37b65f00d337c63"
server_key_version: "encryptionkey@0"
I20260929 20:25:45.101943 31122 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790713339161492-17621-0/minicluster-data/master-0/wal/instance:
uuid: "5ddf8d4c8aee4b1ea7bcdc4c4f712f8f"
format_stamp: "Formatted at 2026-09-29 20:25:45 on dist-test-slave-zbnc"
server_key: "ff7cd6c328f747b4e2ebbe2d6a908137"
server_key_iv: "0884ec8dd99b3a05b37b65f00d337c63"
server_key_version: "encryptionkey@0"
I20260929 20:25:45.107484 31122 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260929 20:25:45.111156 31137 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:25:45.112807 31122 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 20:25:45.113008 31122 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790713339161492-17621-0/minicluster-data/master-0/data,/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790713339161492-17621-0/minicluster-data/master-0/wal
uuid: "5ddf8d4c8aee4b1ea7bcdc4c4f712f8f"
format_stamp: "Formatted at 2026-09-29 20:25:45 on dist-test-slave-zbnc"
server_key: "ff7cd6c328f747b4e2ebbe2d6a908137"
server_key_iv: "0884ec8dd99b3a05b37b65f00d337c63"
server_key_version: "encryptionkey@0"
I20260929 20:25:45.113217 31122 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790713339161492-17621-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790713339161492-17621-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790713339161492-17621-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:25:45.130080 31122 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:25:45.134303 31122 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:25:45.134601 31122 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:25:45.154119 31122 rpc_server.cc:307] RPC server started. Bound to: 127.17.53.126:44735
I20260929 20:25:45.154143 31188 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.53.126:44735 every 8 connection(s)
I20260929 20:25:45.155570 31122 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790713339161492-17621-0/minicluster-data/master-0/data/info.pb
I20260929 20:25:45.159977 31189 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:25:45.161571 17621 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu as pid 31122
I20260929 20:25:45.161810 17621 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790713339161492-17621-0/minicluster-data/master-0/wal/instance
I20260929 20:25:45.162662 17621 external_mini_cluster.cc:1428] Setting key d556fce902dd6d9ec8c1940740baab1d
I20260929 20:25:45.170408 31189 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.53.126" port: 44735 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.17.53.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.53.125@KRBTEST.COM" created.
Sep 29 20:25:45 dist-test-slave-zbnc krb5kdc[31109](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713545, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.53.126@KRBTEST.COM for kudu/127.17.53.126@KRBTEST.COM
I20260929 20:25:45.198151 31192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:25:45.174423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:44735 (local address 127.0.0.1:50536)
0929 20:25:45.175351 (+ 928us) negotiation.cc:107] Waiting for socket to connect
0929 20:25:45.175401 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0929 20:25:45.177537 (+ 2136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:25:45.178399 (+ 862us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:25:45.178405 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:25:45.179162 (+ 757us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:25:45.179512 (+ 350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:25:45.179530 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:25:45.181146 (+ 1616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:25:45.181155 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:25:45.183243 (+ 2088us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:25:45.183260 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:25:45.184251 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:25:45.184260 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:25:45.184474 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:25:45.185194 (+ 720us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:25:45.185229 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:25:45.190197 (+ 4968us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:25:45.193766 (+ 3569us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:25:45.193782 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:25:45.193799 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:25:45.194402 (+ 603us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:25:45.194898 (+ 496us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:25:45.194905 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:25:45.194912 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:25:45.195133 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:25:45.196062 (+ 929us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:25:45.196071 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:25:45.196362 (+ 291us) client_negotiation.cc:769] Sending connection context
0929 20:25:45.196668 (+ 306us) client_negotiation.cc:240] Negotiation successful
0929 20:25:45.197078 (+ 410us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":654,"thread_start_us":423,"threads_started":1}
I20260929 20:25:45.198194 31193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:25:45.175915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50536 (local address 127.17.53.126:44735)
0929 20:25:45.176658 (+ 743us) server_negotiation.cc:206] Beginning negotiation
0929 20:25:45.176666 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 20:25:45.176718 (+ 52us) server_negotiation.cc:407] Connection header received
0929 20:25:45.177931 (+ 1213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:25:45.177937 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:25:45.178173 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:25:45.178387 (+ 214us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:25:45.179707 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:25:45.180937 (+ 1230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:25:45.183392 (+ 2455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:25:45.183981 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:25:45.184115 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:25:45.190498 (+ 6383us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:25:45.190508 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:25:45.190522 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:25:45.190553 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:25:45.193599 (+ 3046us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:25:45.194569 (+ 970us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:25:45.194575 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:25:45.194578 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:25:45.194751 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:25:45.195311 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:25:45.195315 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:25:45.195317 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:25:45.195904 (+ 587us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:25:45.196075 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0929 20:25:45.196880 (+ 805us) server_negotiation.cc:299] Negotiation successful
0929 20:25:45.197289 (+ 409us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":71,"server-negotiator.queue_time_us":527,"thread_start_us":336,"threads_started":1}
I20260929 20:25:45.201695 31189 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.53.125" port: 37523 } has no permanent_uuid. Determining permanent_uuid...
I20260929 20:25:45.203011 31192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:25:45.202511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.125:37523 (local address 127.0.0.1:55254)
0929 20:25:45.202681 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0929 20:25:45.202827 (+ 146us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.53.125:37523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
W20260929 20:25:45.203293 31138 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.53.125:37523: connect: Connection refused (error 111)
I20260929 20:25:45.204995 31192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:25:45.204624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.125:37523 (local address 127.0.0.1:55268)
0929 20:25:45.204782 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0929 20:25:45.204865 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.53.125:37523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260929 20:25:45.205188 31189 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.17.53.125:37523: Network error: Client connection negotiation failed: client connection to 127.17.53.125:37523: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.53.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.125.keytab.
Entry for principal kudu/127.17.53.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.125.keytab.
I20260929 20:25:45.219287 17621 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
/tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790713339161492-17621-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790713339161492-17621-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790713339161492-17621-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790713339161492-17621-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.53.125:37523
--webserver_interface=127.17.53.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.125.keytab
--principal=kudu/127.17.53.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.53.84:36839
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.53.126:44735,127.17.53.125:37523,127.17.53.124:44835
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790713339161492-17621-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790713339161492-17621-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790713339161492-17621-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 20:25:45.254520 31189 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.53.125" port: 37523 } attempt: 1
I20260929 20:25:45.255700 31192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:25:45.255254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.125:37523 (local address 127.0.0.1:55280)
0929 20:25:45.255448 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0929 20:25:45.255560 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.53.125:37523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260929 20:25:45.256731 31192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:25:45.256378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.125:37523 (local address 127.0.0.1:55292)
0929 20:25:45.256523 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0929 20:25:45.256596 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.53.125:37523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260929 20:25:45.256916 31189 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.17.53.125:37523: Network error: Client connection negotiation failed: client connection to 127.17.53.125:37523: connect: Connection refused (error 111)
I20260929 20:25:45.325143 31189 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.53.125" port: 37523 } attempt: 2
I20260929 20:25:45.326349 31192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:25:45.325872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.125:37523 (local address 127.0.0.1:55304)
0929 20:25:45.326106 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0929 20:25:45.326196 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.53.125:37523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260929 20:25:45.327396 31192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:25:45.327029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.125:37523 (local address 127.0.0.1:55308)
0929 20:25:45.327172 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0929 20:25:45.327267 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.53.125:37523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260929 20:25:45.327618 31189 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.17.53.125:37523: Network error: Client connection negotiation failed: client connection to 127.17.53.125:37523: connect: Connection refused (error 111)
I20260929 20:25:45.418848 31189 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.53.125" port: 37523 } attempt: 3
I20260929 20:25:45.420243 31192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:25:45.419739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.125:37523 (local address 127.0.0.1:55318)
0929 20:25:45.419943 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0929 20:25:45.420070 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.53.125:37523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260929 20:25:45.421299 31192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:25:45.420910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.125:37523 (local address 127.0.0.1:55324)
0929 20:25:45.421061 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0929 20:25:45.421155 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.53.125:37523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260929 20:25:45.421490 31189 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.17.53.125:37523: Network error: Client connection negotiation failed: client connection to 127.17.53.125:37523: connect: Connection refused (error 111)
W20260929 20:25:45.555377 31196 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:25:45.555655 31196 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:25:45.555827 31196 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:25:45.562994 31196 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:25:45.563102 31196 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:25:45.563222 31196 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:25:45.563321 31196 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:25:45.563409 31196 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:25:45.564746 31189 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.53.125" port: 37523 } attempt: 4
I20260929 20:25:45.565920 31192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:25:45.565370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.125:37523 (local address 127.0.0.1:55340)
0929 20:25:45.565669 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0929 20:25:45.565778 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.53.125:37523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144}
I20260929 20:25:45.567030 31192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:25:45.566597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.125:37523 (local address 127.0.0.1:55342)
0929 20:25:45.566740 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0929 20:25:45.566840 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.53.125:37523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260929 20:25:45.567229 31189 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.17.53.125:37523: Network error: Client connection negotiation failed: client connection to 127.17.53.125:37523: connect: Connection refused (error 111)
I20260929 20:25:45.573535 31196 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.53.84:36839
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790713339161492-17621-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790713339161492-17621-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.17.53.126:44735,127.17.53.125:37523,127.17.53.124:44835
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.125.keytab
--principal=kudu/127.17.53.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.53.125:37523
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790713339161492-17621-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.17.53.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790713339161492-17621-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:25:45.574931 31196 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:25:45.576297 31196 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:25:45.589138 31204 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:25:45.589236 31202 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:25:45.589311 31201 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:25:45.589833 31196 server_base.cc:1034] running on GCE node
I20260929 20:25:45.590672 31196 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:25:45.592262 31196 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:25:45.593498 31196 hybrid_clock.cc:648] HybridClock initialized: now 1790713545593451 us; error 67 us; skew 500 ppm
I20260929 20:25:45.593869 31196 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:25:45 dist-test-slave-zbnc krb5kdc[31109](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713545, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.53.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:25:45.597306 31196 init.cc:377] Logged in from keytab as kudu/127.17.53.125@KRBTEST.COM (short username kudu)
I20260929 20:25:45.599182 31196 webserver.cc:460] Webserver started at http://127.17.53.125:40767/ using document root <none> and password file <none>
I20260929 20:25:45.599960 31196 fs_manager.cc:359] Metadata directory not provided
I20260929 20:25:45.600128 31196 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:25:45.600482 31196 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:25:45.603991 31196 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790713339161492-17621-0/minicluster-data/master-1/data/instance:
uuid: "6015ffd51e5d409cb73f3cbc6ee13ff9"
format_stamp: "Formatted at 2026-09-29 20:25:45 on dist-test-slave-zbnc"
server_key: "e2c37d731be7786d7e8ecc1f8a410fe5"
server_key_iv: "89be84021ad0db206f3fbd330f96af35"
server_key_version: "encryptionkey@0"
I20260929 20:25:45.604936 31196 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790713339161492-17621-0/minicluster-data/master-1/wal/instance:
uuid: "6015ffd51e5d409cb73f3cbc6ee13ff9"
format_stamp: "Formatted at 2026-09-29 20:25:45 on dist-test-slave-zbnc"
server_key: "e2c37d731be7786d7e8ecc1f8a410fe5"
server_key_iv: "89be84021ad0db206f3fbd330f96af35"
server_key_version: "encryptionkey@0"
I20260929 20:25:45.610622 31196 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260929 20:25:45.614435 31211 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:25:45.615943 31196 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 20:25:45.616127 31196 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790713339161492-17621-0/minicluster-data/master-1/data,/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790713339161492-17621-0/minicluster-data/master-1/wal
uuid: "6015ffd51e5d409cb73f3cbc6ee13ff9"
format_stamp: "Formatted at 2026-09-29 20:25:45 on dist-test-slave-zbnc"
server_key: "e2c37d731be7786d7e8ecc1f8a410fe5"
server_key_iv: "89be84021ad0db206f3fbd330f96af35"
server_key_version: "encryptionkey@0"
I20260929 20:25:45.616356 31196 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790713339161492-17621-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790713339161492-17621-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790713339161492-17621-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:25:45.632560 31196 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:25:45.636938 31196 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:25:45.637241 31196 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:25:45.656373 31196 rpc_server.cc:307] RPC server started. Bound to: 127.17.53.125:37523
I20260929 20:25:45.656415 31262 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.53.125:37523 every 8 connection(s)
I20260929 20:25:45.657801 31196 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790713339161492-17621-0/minicluster-data/master-1/data/info.pb
I20260929 20:25:45.661819 31263 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:25:45.664801 17621 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu as pid 31196
I20260929 20:25:45.665004 17621 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790713339161492-17621-0/minicluster-data/master-1/wal/instance
I20260929 20:25:45.665441 17621 external_mini_cluster.cc:1428] Setting key c8e9575931cd524754a4e635a06b25cf
I20260929 20:25:45.668444 31263 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.53.126" port: 44735 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 20:25:45 dist-test-slave-zbnc krb5kdc[31109](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713545, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.53.125@KRBTEST.COM for kudu/127.17.53.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.17.53.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.53.124@KRBTEST.COM" created.
I20260929 20:25:45.690258 31193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:25:45.671039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50544 (local address 127.17.53.126:44735)
0929 20:25:45.671292 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0929 20:25:45.671301 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 20:25:45.672494 (+ 1193us) server_negotiation.cc:407] Connection header received
0929 20:25:45.674386 (+ 1892us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:25:45.674394 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:25:45.674492 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:25:45.674622 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:25:45.676208 (+ 1586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:25:45.677336 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:25:45.678992 (+ 1656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:25:45.679751 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:25:45.679900 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:25:45.683903 (+ 4003us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:25:45.683909 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:25:45.683913 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:25:45.683947 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:25:45.686631 (+ 2684us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:25:45.687468 (+ 837us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:25:45.687472 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:25:45.687475 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:25:45.687592 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:25:45.688243 (+ 651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:25:45.688250 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:25:45.688253 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:25:45.688571 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:25:45.688684 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0929 20:25:45.689682 (+ 998us) server_negotiation.cc:299] Negotiation successful
0929 20:25:45.689922 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260929 20:25:45.691406 31266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:25:45.671493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:44735 (local address 127.0.0.1:50544)
0929 20:25:45.672308 (+ 815us) negotiation.cc:107] Waiting for socket to connect
0929 20:25:45.672362 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0929 20:25:45.674095 (+ 1733us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:25:45.674819 (+ 724us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:25:45.674828 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:25:45.675444 (+ 616us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:25:45.676015 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:25:45.676031 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:25:45.677507 (+ 1476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:25:45.677512 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:25:45.678778 (+ 1266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:25:45.678790 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:25:45.679970 (+ 1180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:25:45.679975 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:25:45.680172 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:25:45.680985 (+ 813us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:25:45.681015 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:25:45.683640 (+ 2625us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:25:45.686821 (+ 3181us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:25:45.686828 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:25:45.686843 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:25:45.687351 (+ 508us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:25:45.687686 (+ 335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:25:45.687689 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:25:45.687692 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:25:45.687908 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:25:45.688706 (+ 798us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:25:45.688712 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:25:45.689219 (+ 507us) client_negotiation.cc:769] Sending connection context
0929 20:25:45.689566 (+ 347us) client_negotiation.cc:240] Negotiation successful
0929 20:25:45.690109 (+ 543us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":548,"thread_start_us":357,"threads_started":1}
I20260929 20:25:45.693293 31263 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.53.125" port: 37523 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 20:25:45 dist-test-slave-zbnc krb5kdc[31109](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713545, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.53.125@KRBTEST.COM for kudu/127.17.53.125@KRBTEST.COM
I20260929 20:25:45.708176 31266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:25:45.694173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.125:37523 (local address 127.0.0.1:55348)
0929 20:25:45.694357 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0929 20:25:45.694378 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 20:25:45.694478 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:25:45.695542 (+ 1064us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:25:45.695546 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:25:45.696020 (+ 474us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:25:45.696148 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:25:45.696157 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:25:45.697549 (+ 1392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:25:45.697557 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:25:45.698397 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:25:45.698408 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:25:45.699319 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:25:45.699323 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:25:45.699567 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:25:45.700169 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:25:45.700187 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:25:45.702424 (+ 2237us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:25:45.705610 (+ 3186us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:25:45.705616 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:25:45.705620 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:25:45.705970 (+ 350us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:25:45.706345 (+ 375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:25:45.706350 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:25:45.706353 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:25:45.706510 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:25:45.707203 (+ 693us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:25:45.707208 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:25:45.707358 (+ 150us) client_negotiation.cc:769] Sending connection context
0929 20:25:45.707494 (+ 136us) client_negotiation.cc:240] Negotiation successful
0929 20:25:45.707761 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56,"mutex_wait_us":4}
I20260929 20:25:45.708220 31267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:25:45.694432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55348 (local address 127.17.53.125:37523)
0929 20:25:45.695121 (+ 689us) server_negotiation.cc:206] Beginning negotiation
0929 20:25:45.695128 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:25:45.695152 (+ 24us) server_negotiation.cc:407] Connection header received
0929 20:25:45.695196 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:25:45.695202 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:25:45.695398 (+ 196us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:25:45.695562 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:25:45.696305 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:25:45.697426 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:25:45.698534 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:25:45.699172 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:25:45.699310 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:25:45.702634 (+ 3324us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:25:45.702645 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:25:45.702658 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:25:45.702685 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:25:45.705466 (+ 2781us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:25:45.706099 (+ 633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:25:45.706108 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:25:45.706114 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:25:45.706227 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:25:45.706617 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:25:45.706624 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:25:45.706627 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:25:45.707068 (+ 441us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:25:45.707285 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0929 20:25:45.707638 (+ 353us) server_negotiation.cc:299] Negotiation successful
0929 20:25:45.707858 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":46,"server-negotiator.queue_time_us":543,"thread_start_us":354,"threads_started":1}
I20260929 20:25:45.710678 31263 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.53.124" port: 44835 } has no permanent_uuid. Determining permanent_uuid...
I20260929 20:25:45.711894 31266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:25:45.711326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.124:44835 (local address 127.0.0.1:36606)
0929 20:25:45.711572 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0929 20:25:45.711715 (+ 143us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.53.124:44835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
W20260929 20:25:45.712128 31213 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.53.124:44835: connect: Connection refused (error 111)
I20260929 20:25:45.713627 31266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:25:45.713281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.124:44835 (local address 127.0.0.1:36610)
0929 20:25:45.713379 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0929 20:25:45.713467 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.53.124:44835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260929 20:25:45.713850 31263 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.17.53.124:44835: Network error: Client connection negotiation failed: client connection to 127.17.53.124:44835: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.53.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.124.keytab.
Entry for principal kudu/127.17.53.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.124.keytab.
I20260929 20:25:45.718586 17621 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
/tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790713339161492-17621-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790713339161492-17621-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790713339161492-17621-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790713339161492-17621-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.53.124:44835
--webserver_interface=127.17.53.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.124.keytab
--principal=kudu/127.17.53.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.53.84:36839
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.53.126:44735,127.17.53.125:37523,127.17.53.124:44835
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790713339161492-17621-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790713339161492-17621-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790713339161492-17621-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 20:25:45.763172 31263 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.53.124" port: 44835 } attempt: 1
I20260929 20:25:45.764523 31266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:25:45.764060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.124:44835 (local address 127.0.0.1:36618)
0929 20:25:45.764271 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0929 20:25:45.764376 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.53.124:44835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260929 20:25:45.765570 31266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:25:45.765249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.124:44835 (local address 127.0.0.1:36630)
0929 20:25:45.765403 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0929 20:25:45.765469 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.53.124:44835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260929 20:25:45.765834 31263 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.17.53.124:44835: Network error: Client connection negotiation failed: client connection to 127.17.53.124:44835: connect: Connection refused (error 111)
I20260929 20:25:45.834093 31263 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.53.124" port: 44835 } attempt: 2
I20260929 20:25:45.835379 31266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:25:45.834840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.124:44835 (local address 127.0.0.1:36642)
0929 20:25:45.835073 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0929 20:25:45.835170 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.53.124:44835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109}
I20260929 20:25:45.836372 31266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:25:45.836031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.124:44835 (local address 127.0.0.1:36658)
0929 20:25:45.836180 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0929 20:25:45.836260 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.53.124:44835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260929 20:25:45.836575 31263 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.17.53.124:44835: Network error: Client connection negotiation failed: client connection to 127.17.53.124:44835: connect: Connection refused (error 111)
I20260929 20:25:45.866528 31189 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.53.125" port: 37523 } attempt: 5
Sep 29 20:25:45 dist-test-slave-zbnc krb5kdc[31109](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713545, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.53.126@KRBTEST.COM for kudu/127.17.53.125@KRBTEST.COM
I20260929 20:25:45.880002 31267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:25:45.867440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55354 (local address 127.17.53.125:37523)
0929 20:25:45.867619 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0929 20:25:45.867627 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 20:25:45.867650 (+ 23us) server_negotiation.cc:407] Connection header received
0929 20:25:45.867894 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:25:45.867899 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:25:45.867978 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:25:45.868087 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:25:45.868908 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:25:45.869618 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:25:45.870781 (+ 1163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:25:45.871332 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:25:45.871466 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:25:45.874657 (+ 3191us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:25:45.874662 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:25:45.874665 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:25:45.874697 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:25:45.877329 (+ 2632us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:25:45.877985 (+ 656us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:25:45.877991 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:25:45.877995 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:25:45.878117 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:25:45.878502 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:25:45.878508 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:25:45.878511 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:25:45.878805 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:25:45.879040 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0929 20:25:45.879433 (+ 393us) server_negotiation.cc:299] Negotiation successful
0929 20:25:45.879657 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260929 20:25:45.880002 31192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:25:45.867328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.125:37523 (local address 127.0.0.1:55354)
0929 20:25:45.867555 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0929 20:25:45.867577 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 20:25:45.867723 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:25:45.868121 (+ 398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:25:45.868126 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:25:45.868545 (+ 419us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:25:45.868735 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:25:45.868744 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:25:45.869765 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:25:45.869771 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:25:45.870638 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:25:45.870649 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:25:45.871491 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:25:45.871498 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:25:45.871693 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:25:45.872432 (+ 739us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:25:45.872450 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:25:45.874461 (+ 2011us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:25:45.877475 (+ 3014us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:25:45.877481 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:25:45.877486 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:25:45.877863 (+ 377us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:25:45.878232 (+ 369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:25:45.878237 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:25:45.878240 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:25:45.878390 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:25:45.879008 (+ 618us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:25:45.879013 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:25:45.879171 (+ 158us) client_negotiation.cc:769] Sending connection context
0929 20:25:45.879329 (+ 158us) client_negotiation.cc:240] Negotiation successful
0929 20:25:45.879633 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":73}
I20260929 20:25:45.881392 31189 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.53.124" port: 44835 } has no permanent_uuid. Determining permanent_uuid...
I20260929 20:25:45.882480 31192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:25:45.881991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.124:44835 (local address 127.0.0.1:36670)
0929 20:25:45.882191 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0929 20:25:45.882301 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.53.124:44835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260929 20:25:45.883489 31192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:25:45.883115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.124:44835 (local address 127.0.0.1:36674)
0929 20:25:45.883266 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0929 20:25:45.883357 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.53.124:44835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260929 20:25:45.883706 31189 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.17.53.124:44835: Network error: Client connection negotiation failed: client connection to 127.17.53.124:44835: connect: Connection refused (error 111)
I20260929 20:25:45.927796 31263 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.53.124" port: 44835 } attempt: 3
I20260929 20:25:45.928942 31266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:25:45.928465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.124:44835 (local address 127.0.0.1:36686)
0929 20:25:45.928668 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0929 20:25:45.928780 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.53.124:44835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260929 20:25:45.930011 31266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:25:45.929596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.124:44835 (local address 127.0.0.1:36694)
0929 20:25:45.929823 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0929 20:25:45.929897 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.53.124:44835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116}
W20260929 20:25:45.930209 31263 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.17.53.124:44835: Network error: Client connection negotiation failed: client connection to 127.17.53.124:44835: connect: Connection refused (error 111)
I20260929 20:25:45.934950 31189 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.53.124" port: 44835 } attempt: 1
I20260929 20:25:45.936105 31192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:25:45.935505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.124:44835 (local address 127.0.0.1:36698)
0929 20:25:45.935755 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0929 20:25:45.935877 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.53.124:44835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260929 20:25:45.937219 31192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:25:45.936825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.124:44835 (local address 127.0.0.1:36710)
0929 20:25:45.937002 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0929 20:25:45.937099 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.53.124:44835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260929 20:25:45.937443 31189 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.17.53.124:44835: Network error: Client connection negotiation failed: client connection to 127.17.53.124:44835: connect: Connection refused (error 111)
I20260929 20:25:46.005688 31189 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.53.124" port: 44835 } attempt: 2
I20260929 20:25:46.007089 31192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:25:46.006474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.124:44835 (local address 127.0.0.1:36726)
0929 20:25:46.006715 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0929 20:25:46.006869 (+ 154us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.53.124:44835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260929 20:25:46.008317 31192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:25:46.007925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.124:44835 (local address 127.0.0.1:36728)
0929 20:25:46.008081 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0929 20:25:46.008177 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.53.124:44835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260929 20:25:46.008525 31189 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.17.53.124:44835: Network error: Client connection negotiation failed: client connection to 127.17.53.124:44835: connect: Connection refused (error 111)
W20260929 20:25:46.055490 31270 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:25:46.055773 31270 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:25:46.055860 31270 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:25:46.063274 31270 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:25:46.063386 31270 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:25:46.063432 31270 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:25:46.063488 31270 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:25:46.063531 31270 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:25:46.073307 31270 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.53.84:36839
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790713339161492-17621-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790713339161492-17621-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.17.53.126:44735,127.17.53.125:37523,127.17.53.124:44835
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.124.keytab
--principal=kudu/127.17.53.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.53.124:44835
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790713339161492-17621-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.17.53.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790713339161492-17621-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:25:46.073458 31263 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.53.124" port: 44835 } attempt: 4
I20260929 20:25:46.074622 31266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:25:46.074127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.124:44835 (local address 127.0.0.1:36732)
0929 20:25:46.074378 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0929 20:25:46.074462 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.53.124:44835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260929 20:25:46.074757 31270 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:25:46.075649 31266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:25:46.075352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.124:44835 (local address 127.0.0.1:36734)
0929 20:25:46.075482 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0929 20:25:46.075549 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.53.124:44835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260929 20:25:46.075845 31263 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.17.53.124:44835: Network error: Client connection negotiation failed: client connection to 127.17.53.124:44835: connect: Connection refused (error 111)
I20260929 20:25:46.076362 31270 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:25:46.088284 31275 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:25:46.088374 31276 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:25:46.088791 31278 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:25:46.089041 31270 server_base.cc:1034] running on GCE node
I20260929 20:25:46.089812 31270 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:25:46.091374 31270 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:25:46.092617 31270 hybrid_clock.cc:648] HybridClock initialized: now 1790713546092581 us; error 31 us; skew 500 ppm
I20260929 20:25:46.092983 31270 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:25:46 dist-test-slave-zbnc krb5kdc[31109](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713546, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.53.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:25:46.096278 31270 init.cc:377] Logged in from keytab as kudu/127.17.53.124@KRBTEST.COM (short username kudu)
I20260929 20:25:46.097801 31270 webserver.cc:460] Webserver started at http://127.17.53.124:34035/ using document root <none> and password file <none>
I20260929 20:25:46.098562 31270 fs_manager.cc:359] Metadata directory not provided
I20260929 20:25:46.098693 31270 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:25:46.099044 31270 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:25:46.102070 31270 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790713339161492-17621-0/minicluster-data/master-2/data/instance:
uuid: "38800a96109d41829eb37e95ff7286f7"
format_stamp: "Formatted at 2026-09-29 20:25:46 on dist-test-slave-zbnc"
server_key: "1fd3304c77f551a2ac441fd16c6acfa5"
server_key_iv: "da0b63245cfe0e6adfd620fd256ce64a"
server_key_version: "encryptionkey@0"
I20260929 20:25:46.102950 31270 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790713339161492-17621-0/minicluster-data/master-2/wal/instance:
uuid: "38800a96109d41829eb37e95ff7286f7"
format_stamp: "Formatted at 2026-09-29 20:25:46 on dist-test-slave-zbnc"
server_key: "1fd3304c77f551a2ac441fd16c6acfa5"
server_key_iv: "da0b63245cfe0e6adfd620fd256ce64a"
server_key_version: "encryptionkey@0"
I20260929 20:25:46.108335 31270 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.002s
I20260929 20:25:46.111932 31285 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:25:46.113399 31270 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 20:25:46.113607 31270 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790713339161492-17621-0/minicluster-data/master-2/data,/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790713339161492-17621-0/minicluster-data/master-2/wal
uuid: "38800a96109d41829eb37e95ff7286f7"
format_stamp: "Formatted at 2026-09-29 20:25:46 on dist-test-slave-zbnc"
server_key: "1fd3304c77f551a2ac441fd16c6acfa5"
server_key_iv: "da0b63245cfe0e6adfd620fd256ce64a"
server_key_version: "encryptionkey@0"
I20260929 20:25:46.113837 31270 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790713339161492-17621-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790713339161492-17621-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790713339161492-17621-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:25:46.114750 31189 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.53.124" port: 44835 } attempt: 3
I20260929 20:25:46.115931 31192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:25:46.115443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.124:44835 (local address 127.0.0.1:36740)
0929 20:25:46.115653 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0929 20:25:46.115756 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.53.124:44835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260929 20:25:46.116971 31192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:25:46.116599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.124:44835 (local address 127.0.0.1:36754)
0929 20:25:46.116809 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0929 20:25:46.116875 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.53.124:44835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
W20260929 20:25:46.117213 31189 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.17.53.124:44835: Network error: Client connection negotiation failed: client connection to 127.17.53.124:44835: connect: Connection refused (error 111)
I20260929 20:25:46.127816 31270 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:25:46.132175 31270 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:25:46.132479 31270 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:25:46.151214 31270 rpc_server.cc:307] RPC server started. Bound to: 127.17.53.124:44835
I20260929 20:25:46.151216 31336 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.53.124:44835 every 8 connection(s)
I20260929 20:25:46.152661 31270 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790713339161492-17621-0/minicluster-data/master-2/data/info.pb
I20260929 20:25:46.157312 31337 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:25:46.160805 17621 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu as pid 31270
I20260929 20:25:46.161055 17621 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790713339161492-17621-0/minicluster-data/master-2/wal/instance
I20260929 20:25:46.161593 17621 external_mini_cluster.cc:1428] Setting key 35f91a665ddf7b88866e35fb4640e58f
I20260929 20:25:46.166393 31337 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.53.126" port: 44735 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 20:25:46 dist-test-slave-zbnc krb5kdc[31109](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713546, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.53.124@KRBTEST.COM for kudu/127.17.53.126@KRBTEST.COM
Sep 29 20:25:46 dist-test-slave-zbnc krb5kdc[31109](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713544, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.53.124@KRBTEST.COM
I20260929 20:25:46.187079 31193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:25:46.169223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50546 (local address 127.17.53.126:44735)
0929 20:25:46.169450 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0929 20:25:46.169456 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:25:46.170281 (+ 825us) server_negotiation.cc:407] Connection header received
0929 20:25:46.170563 (+ 282us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:25:46.170568 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:25:46.170644 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:25:46.170743 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:25:46.172502 (+ 1759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:25:46.173616 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:25:46.175736 (+ 2120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:25:46.176304 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:25:46.176438 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:25:46.180451 (+ 4013us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:25:46.180458 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:25:46.180465 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:25:46.180511 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:25:46.183587 (+ 3076us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:25:46.184484 (+ 897us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:25:46.184488 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:25:46.184490 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:25:46.184617 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:25:46.185078 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:25:46.185084 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:25:46.185087 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:25:46.185434 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:25:46.185594 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0929 20:25:46.186442 (+ 848us) server_negotiation.cc:299] Negotiation successful
0929 20:25:46.186655 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260929 20:25:46.188459 31341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:25:46.169259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:44735 (local address 127.0.0.1:50546)
0929 20:25:46.170118 (+ 859us) negotiation.cc:107] Waiting for socket to connect
0929 20:25:46.170160 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 20:25:46.170331 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:25:46.170843 (+ 512us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:25:46.170877 (+ 34us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:25:46.171714 (+ 837us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:25:46.172301 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:25:46.172327 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:25:46.173789 (+ 1462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:25:46.173793 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:25:46.175599 (+ 1806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:25:46.175615 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:25:46.176525 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:25:46.176530 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:25:46.176684 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:25:46.177407 (+ 723us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:25:46.177441 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:25:46.180222 (+ 2781us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:25:46.183731 (+ 3509us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:25:46.183740 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:25:46.183763 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:25:46.184363 (+ 600us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:25:46.184722 (+ 359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:25:46.184727 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:25:46.184730 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:25:46.184957 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:25:46.185554 (+ 597us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:25:46.185562 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:25:46.185989 (+ 427us) client_negotiation.cc:769] Sending connection context
0929 20:25:46.186339 (+ 350us) client_negotiation.cc:240] Negotiation successful
0929 20:25:46.186849 (+ 510us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":580,"thread_start_us":442,"threads_started":1}
I20260929 20:25:46.188477 31340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:25:46.167177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36760 (local address 127.17.53.124:44835)
0929 20:25:46.167938 (+ 761us) server_negotiation.cc:206] Beginning negotiation
0929 20:25:46.167949 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 20:25:46.167988 (+ 39us) server_negotiation.cc:407] Connection header received
0929 20:25:46.169875 (+ 1887us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:25:46.169897 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:25:46.170310 (+ 413us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:25:46.170599 (+ 289us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:25:46.172507 (+ 1908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:25:46.174104 (+ 1597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:25:46.176014 (+ 1910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:25:46.176750 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:25:46.176870 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:25:46.181521 (+ 4651us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:25:46.181529 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:25:46.181541 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:25:46.181568 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:25:46.184733 (+ 3165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:25:46.185554 (+ 821us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:25:46.185562 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:25:46.185565 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:25:46.185714 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:25:46.186081 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:25:46.186086 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:25:46.186089 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:25:46.186504 (+ 415us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:25:46.186774 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0929 20:25:46.187214 (+ 440us) server_negotiation.cc:299] Negotiation successful
0929 20:25:46.187464 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":563,"thread_start_us":339,"threads_started":1}
I20260929 20:25:46.190237 31337 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.53.125" port: 37523 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 20:25:46 dist-test-slave-zbnc krb5kdc[31109](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713546, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.53.124@KRBTEST.COM for kudu/127.17.53.125@KRBTEST.COM
Sep 29 20:25:46 dist-test-slave-zbnc krb5kdc[31109](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713544, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.53.126@KRBTEST.COM
I20260929 20:25:46.205746 31341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:25:46.191368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.125:37523 (local address 127.0.0.1:55364)
0929 20:25:46.191553 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0929 20:25:46.191572 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 20:25:46.191697 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:25:46.192098 (+ 401us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:25:46.192102 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:25:46.192560 (+ 458us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:25:46.192756 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:25:46.192770 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:25:46.194278 (+ 1508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:25:46.194283 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:25:46.195708 (+ 1425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:25:46.195720 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:25:46.196637 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:25:46.196641 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:25:46.196816 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:25:46.197425 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:25:46.197443 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:25:46.199694 (+ 2251us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:25:46.203292 (+ 3598us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:25:46.203299 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:25:46.203306 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:25:46.203724 (+ 418us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:25:46.204133 (+ 409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:25:46.204137 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:25:46.204139 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:25:46.204376 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:25:46.205008 (+ 632us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:25:46.205011 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:25:46.205145 (+ 134us) client_negotiation.cc:769] Sending connection context
0929 20:25:46.205288 (+ 143us) client_negotiation.cc:240] Negotiation successful
0929 20:25:46.205499 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":63}
I20260929 20:25:46.206262 31267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:25:46.191157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55364 (local address 127.17.53.125:37523)
0929 20:25:46.191423 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0929 20:25:46.191431 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 20:25:46.191661 (+ 230us) server_negotiation.cc:407] Connection header received
0929 20:25:46.191859 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:25:46.191866 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:25:46.191957 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:25:46.192077 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:25:46.192977 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:25:46.194115 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:25:46.195842 (+ 1727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:25:46.196499 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:25:46.196622 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:25:46.199898 (+ 3276us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:25:46.199906 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:25:46.199914 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:25:46.199966 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:25:46.203141 (+ 3175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:25:46.203856 (+ 715us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:25:46.203862 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:25:46.203867 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:25:46.204020 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:25:46.204509 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:25:46.204513 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:25:46.204516 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:25:46.204890 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:25:46.205050 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0929 20:25:46.205451 (+ 401us) server_negotiation.cc:299] Negotiation successful
0929 20:25:46.205672 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260929 20:25:46.207314 31337 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.53.124" port: 44835 } has no permanent_uuid. Determining permanent_uuid...
I20260929 20:25:46.209195 31193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:25:46.192168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50560 (local address 127.17.53.126:44735)
0929 20:25:46.192351 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0929 20:25:46.192356 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:25:46.192374 (+ 18us) server_negotiation.cc:407] Connection header received
0929 20:25:46.192536 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:25:46.192541 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:25:46.192614 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:25:46.192722 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:25:46.194283 (+ 1561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:25:46.195446 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:25:46.197119 (+ 1673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:25:46.197542 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:25:46.197673 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:25:46.202499 (+ 4826us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:25:46.202507 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:25:46.202512 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:25:46.202553 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:25:46.206036 (+ 3483us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:25:46.206902 (+ 866us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:25:46.206909 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:25:46.206913 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:25:46.207081 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:25:46.207575 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:25:46.207580 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:25:46.207584 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:25:46.207920 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:25:46.208177 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0929 20:25:46.208823 (+ 646us) server_negotiation.cc:299] Negotiation successful
0929 20:25:46.208980 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
Sep 29 20:25:46 dist-test-slave-zbnc krb5kdc[31109](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713546, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.53.124@KRBTEST.COM for kudu/127.17.53.124@KRBTEST.COM
Sep 29 20:25:46 dist-test-slave-zbnc krb5kdc[31109](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713544, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.53.125@KRBTEST.COM
I20260929 20:25:46.221927 31340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:25:46.208199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36776 (local address 127.17.53.124:44835)
0929 20:25:46.208335 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0929 20:25:46.208339 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:25:46.208352 (+ 13us) server_negotiation.cc:407] Connection header received
0929 20:25:46.208480 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:25:46.208483 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:25:46.208534 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:25:46.208604 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:25:46.209691 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:25:46.210644 (+ 953us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:25:46.212297 (+ 1653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:25:46.212737 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:25:46.212834 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:25:46.216089 (+ 3255us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:25:46.216094 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:25:46.216097 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:25:46.216123 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:25:46.219089 (+ 2966us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:25:46.219844 (+ 755us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:25:46.219848 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:25:46.219850 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:25:46.220022 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:25:46.220491 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:25:46.220494 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:25:46.220496 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:25:46.220784 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:25:46.220893 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0929 20:25:46.221385 (+ 492us) server_negotiation.cc:299] Negotiation successful
0929 20:25:46.221610 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260929 20:25:46.222285 31341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:25:46.208060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.124:44835 (local address 127.0.0.1:36776)
0929 20:25:46.208245 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0929 20:25:46.208265 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 20:25:46.208375 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:25:46.208670 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:25:46.208678 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:25:46.209296 (+ 618us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:25:46.209512 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:25:46.209521 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:25:46.210765 (+ 1244us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:25:46.210771 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:25:46.212141 (+ 1370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:25:46.212158 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:25:46.212898 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:25:46.212905 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:25:46.213113 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:25:46.213715 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:25:46.213732 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:25:46.215920 (+ 2188us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:25:46.219218 (+ 3298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:25:46.219231 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:25:46.219235 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:25:46.219733 (+ 498us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:25:46.220207 (+ 474us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:25:46.220212 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:25:46.220217 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:25:46.220385 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:25:46.220909 (+ 524us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:25:46.220917 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:25:46.221087 (+ 170us) client_negotiation.cc:769] Sending connection context
0929 20:25:46.221282 (+ 195us) client_negotiation.cc:240] Negotiation successful
0929 20:25:46.221799 (+ 517us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":66,"mutex_wait_us":94}
I20260929 20:25:46.226709 31267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:25:46.211219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55374 (local address 127.17.53.125:37523)
0929 20:25:46.211373 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0929 20:25:46.211377 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:25:46.211395 (+ 18us) server_negotiation.cc:407] Connection header received
0929 20:25:46.211586 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:25:46.211589 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:25:46.211644 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:25:46.211722 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:25:46.213132 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:25:46.213901 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:25:46.215571 (+ 1670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:25:46.216004 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:25:46.216137 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:25:46.221021 (+ 4884us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:25:46.221025 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:25:46.221028 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:25:46.221058 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:25:46.224133 (+ 3075us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:25:46.224794 (+ 661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:25:46.224799 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:25:46.224803 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:25:46.224942 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:25:46.225326 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:25:46.225332 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:25:46.225335 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:25:46.225602 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:25:46.225896 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0929 20:25:46.226181 (+ 285us) server_negotiation.cc:299] Negotiation successful
0929 20:25:46.226403 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 20:25:46.227133 31337 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 38800a96109d41829eb37e95ff7286f7: Bootstrap starting.
I20260929 20:25:46.230751 31337 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 38800a96109d41829eb37e95ff7286f7: Neither blocks nor log segments found. Creating new log.
I20260929 20:25:46.231463 17621 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 20:25:46.231918 31337 log.cc:824] T 00000000000000000000000000000000 P 38800a96109d41829eb37e95ff7286f7: Log is configured to *not* fsync() on all Append() calls
I20260929 20:25:46.234429 31337 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 38800a96109d41829eb37e95ff7286f7: No bootstrap required, opened a new log
I20260929 20:25:46.248212 31337 raft_consensus.cc:348] T 00000000000000000000000000000000 P 38800a96109d41829eb37e95ff7286f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ddf8d4c8aee4b1ea7bcdc4c4f712f8f" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 44735 } } peers { permanent_uuid: "6015ffd51e5d409cb73f3cbc6ee13ff9" member_type: VOTER last_known_addr { host: "127.17.53.125" port: 37523 } } peers { permanent_uuid: "38800a96109d41829eb37e95ff7286f7" member_type: VOTER last_known_addr { host: "127.17.53.124" port: 44835 } }
I20260929 20:25:46.248832 31337 raft_consensus.cc:374] T 00000000000000000000000000000000 P 38800a96109d41829eb37e95ff7286f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:25:46.249101 31337 raft_consensus.cc:729] T 00000000000000000000000000000000 P 38800a96109d41829eb37e95ff7286f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 38800a96109d41829eb37e95ff7286f7, State: Initialized, Role: FOLLOWER
I20260929 20:25:46.250522 31337 consensus_queue.cc:260] T 00000000000000000000000000000000 P 38800a96109d41829eb37e95ff7286f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ddf8d4c8aee4b1ea7bcdc4c4f712f8f" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 44735 } } peers { permanent_uuid: "6015ffd51e5d409cb73f3cbc6ee13ff9" member_type: VOTER last_known_addr { host: "127.17.53.125" port: 37523 } } peers { permanent_uuid: "38800a96109d41829eb37e95ff7286f7" member_type: VOTER last_known_addr { host: "127.17.53.124" port: 44835 } }
I20260929 20:25:46.251611 31349 sys_catalog.cc:455] T 00000000000000000000000000000000 P 38800a96109d41829eb37e95ff7286f7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ddf8d4c8aee4b1ea7bcdc4c4f712f8f" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 44735 } } peers { permanent_uuid: "6015ffd51e5d409cb73f3cbc6ee13ff9" member_type: VOTER last_known_addr { host: "127.17.53.125" port: 37523 } } peers { permanent_uuid: "38800a96109d41829eb37e95ff7286f7" member_type: VOTER last_known_addr { host: "127.17.53.124" port: 44835 } } }
I20260929 20:25:46.252002 31349 sys_catalog.cc:458] T 00000000000000000000000000000000 P 38800a96109d41829eb37e95ff7286f7 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 20:25:46.253388 31337 sys_catalog.cc:564] T 00000000000000000000000000000000 P 38800a96109d41829eb37e95ff7286f7 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:25:46.270728 31267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:25:46.237056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55382 (local address 127.17.53.125:37523)
0929 20:25:46.237312 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0929 20:25:46.237319 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:25:46.246971 (+ 9652us) server_negotiation.cc:407] Connection header received
0929 20:25:46.247200 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:25:46.247210 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:25:46.247334 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:25:46.247453 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:25:46.250524 (+ 3071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:25:46.251969 (+ 1445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:25:46.254167 (+ 2198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:25:46.254908 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:25:46.255021 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:25:46.259830 (+ 4809us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:25:46.259839 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:25:46.259847 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:25:46.259903 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:25:46.266295 (+ 6392us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:25:46.267286 (+ 991us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:25:46.267292 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:25:46.267295 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:25:46.267486 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:25:46.268019 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:25:46.268026 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:25:46.268029 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:25:46.268453 (+ 424us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:25:46.268716 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0929 20:25:46.270023 (+ 1307us) server_negotiation.cc:299] Negotiation successful
0929 20:25:46.270285 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260929 20:25:46.269515 31270 master_runner.cc:186] Error getting master registration for 127.17.53.126:44735: OK, instance_id { permanent_uuid: "5ddf8d4c8aee4b1ea7bcdc4c4f712f8f" instance_seqno: 1790713545148840 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
W20260929 20:25:46.281061 31361 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 38800a96109d41829eb37e95ff7286f7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 20:25:46.281193 31361 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 20:25:46.281203 31270 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 20:25:46.282930 31193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:25:46.237356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50564 (local address 127.17.53.126:44735)
0929 20:25:46.237640 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0929 20:25:46.237649 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 20:25:46.252562 (+ 14913us) server_negotiation.cc:407] Connection header received
0929 20:25:46.253930 (+ 1368us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:25:46.253937 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:25:46.254023 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:25:46.254100 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:25:46.258419 (+ 4319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:25:46.259968 (+ 1549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:25:46.262453 (+ 2485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:25:46.263308 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:25:46.263472 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:25:46.271452 (+ 7980us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:25:46.271463 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:25:46.271472 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:25:46.271851 (+ 379us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:25:46.278439 (+ 6588us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:25:46.279397 (+ 958us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:25:46.279403 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:25:46.279406 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:25:46.279582 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:25:46.280110 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:25:46.280118 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:25:46.280122 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:25:46.280517 (+ 395us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:25:46.281740 (+ 1223us) server_negotiation.cc:1035] Waiting for connection context
0929 20:25:46.281851 (+ 111us) server_negotiation.cc:299] Negotiation successful
0929 20:25:46.282484 (+ 633us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260929 20:25:46.283587 31340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:25:46.253172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36780 (local address 127.17.53.124:44835)
0929 20:25:46.253339 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0929 20:25:46.253345 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:25:46.259874 (+ 6529us) server_negotiation.cc:407] Connection header received
0929 20:25:46.260449 (+ 575us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:25:46.260457 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:25:46.260554 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:25:46.260647 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:25:46.262937 (+ 2290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:25:46.264417 (+ 1480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:25:46.266655 (+ 2238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:25:46.267445 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:25:46.267564 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:25:46.275617 (+ 8053us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:25:46.275631 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:25:46.275639 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:25:46.275693 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:25:46.279888 (+ 4195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:25:46.280957 (+ 1069us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:25:46.280965 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:25:46.280984 (+ 19us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:25:46.281190 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:25:46.281699 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:25:46.281706 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:25:46.281711 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:25:46.282170 (+ 459us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:25:46.282447 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0929 20:25:46.283067 (+ 620us) server_negotiation.cc:299] Negotiation successful
0929 20:25:46.283272 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 20:25:46.294426 31189 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.53.124" port: 44835 } attempt: 4
Sep 29 20:25:46 dist-test-slave-zbnc krb5kdc[31109](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713545, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.53.126@KRBTEST.COM for kudu/127.17.53.124@KRBTEST.COM
I20260929 20:25:46.309046 31192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:25:46.295009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.124:44835 (local address 127.0.0.1:36784)
0929 20:25:46.295202 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0929 20:25:46.295266 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0929 20:25:46.295388 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:25:46.295824 (+ 436us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:25:46.295828 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:25:46.296344 (+ 516us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:25:46.296581 (+ 237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:25:46.296596 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:25:46.297846 (+ 1250us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:25:46.297852 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:25:46.298986 (+ 1134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:25:46.299003 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:25:46.299747 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:25:46.299753 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:25:46.299949 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:25:46.300616 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:25:46.300633 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:25:46.302661 (+ 2028us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:25:46.306493 (+ 3832us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:25:46.306500 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:25:46.306504 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:25:46.306920 (+ 416us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:25:46.307319 (+ 399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:25:46.307325 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:25:46.307329 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:25:46.307469 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:25:46.308033 (+ 564us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:25:46.308037 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:25:46.308210 (+ 173us) client_negotiation.cc:769] Sending connection context
0929 20:25:46.308360 (+ 150us) client_negotiation.cc:240] Negotiation successful
0929 20:25:46.308645 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55}
I20260929 20:25:46.309044 31340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:25:46.295087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36784 (local address 127.17.53.124:44835)
0929 20:25:46.295281 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0929 20:25:46.295289 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 20:25:46.295311 (+ 22us) server_negotiation.cc:407] Connection header received
0929 20:25:46.295564 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:25:46.295572 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:25:46.295651 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:25:46.296378 (+ 727us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:25:46.296767 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:25:46.297706 (+ 939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:25:46.299147 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:25:46.299595 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:25:46.299734 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:25:46.302897 (+ 3163us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:25:46.302904 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:25:46.302909 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:25:46.302954 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:25:46.306320 (+ 3366us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:25:46.307036 (+ 716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:25:46.307042 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:25:46.307046 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:25:46.307203 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:25:46.307584 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:25:46.307589 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:25:46.307592 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:25:46.307899 (+ 307us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:25:46.308056 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0929 20:25:46.308459 (+ 403us) server_negotiation.cc:299] Negotiation successful
0929 20:25:46.308669 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260929 20:25:46.313637 31189 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5ddf8d4c8aee4b1ea7bcdc4c4f712f8f: Bootstrap starting.
I20260929 20:25:46.317955 31189 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5ddf8d4c8aee4b1ea7bcdc4c4f712f8f: Neither blocks nor log segments found. Creating new log.
I20260929 20:25:46.319219 31189 log.cc:824] T 00000000000000000000000000000000 P 5ddf8d4c8aee4b1ea7bcdc4c4f712f8f: Log is configured to *not* fsync() on all Append() calls
I20260929 20:25:46.322000 31189 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5ddf8d4c8aee4b1ea7bcdc4c4f712f8f: No bootstrap required, opened a new log
I20260929 20:25:46.330013 31189 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5ddf8d4c8aee4b1ea7bcdc4c4f712f8f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ddf8d4c8aee4b1ea7bcdc4c4f712f8f" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 44735 } } peers { permanent_uuid: "6015ffd51e5d409cb73f3cbc6ee13ff9" member_type: VOTER last_known_addr { host: "127.17.53.125" port: 37523 } } peers { permanent_uuid: "38800a96109d41829eb37e95ff7286f7" member_type: VOTER last_known_addr { host: "127.17.53.124" port: 44835 } }
I20260929 20:25:46.330386 31189 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5ddf8d4c8aee4b1ea7bcdc4c4f712f8f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:25:46.330518 31189 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5ddf8d4c8aee4b1ea7bcdc4c4f712f8f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ddf8d4c8aee4b1ea7bcdc4c4f712f8f, State: Initialized, Role: FOLLOWER
I20260929 20:25:46.331172 31189 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5ddf8d4c8aee4b1ea7bcdc4c4f712f8f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ddf8d4c8aee4b1ea7bcdc4c4f712f8f" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 44735 } } peers { permanent_uuid: "6015ffd51e5d409cb73f3cbc6ee13ff9" member_type: VOTER last_known_addr { host: "127.17.53.125" port: 37523 } } peers { permanent_uuid: "38800a96109d41829eb37e95ff7286f7" member_type: VOTER last_known_addr { host: "127.17.53.124" port: 44835 } }
I20260929 20:25:46.332182 31364 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5ddf8d4c8aee4b1ea7bcdc4c4f712f8f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ddf8d4c8aee4b1ea7bcdc4c4f712f8f" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 44735 } } peers { permanent_uuid: "6015ffd51e5d409cb73f3cbc6ee13ff9" member_type: VOTER last_known_addr { host: "127.17.53.125" port: 37523 } } peers { permanent_uuid: "38800a96109d41829eb37e95ff7286f7" member_type: VOTER last_known_addr { host: "127.17.53.124" port: 44835 } } }
I20260929 20:25:46.332640 31364 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5ddf8d4c8aee4b1ea7bcdc4c4f712f8f [sys.catalog]: This master's current role is: FOLLOWER
I20260929 20:25:46.333674 31189 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5ddf8d4c8aee4b1ea7bcdc4c4f712f8f [sys.catalog]: configured and running, proceeding with master startup.
W20260929 20:25:46.344799 31375 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5ddf8d4c8aee4b1ea7bcdc4c4f712f8f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 20:25:46.345013 31375 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 20:25:46.343056 31122 master_runner.cc:186] Error getting master registration for 127.17.53.125:37523: OK, instance_id { permanent_uuid: "6015ffd51e5d409cb73f3cbc6ee13ff9" instance_seqno: 1790713545651024 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 20:25:46.355401 31122 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 20:25:46.375063 31263 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.53.124" port: 44835 } attempt: 5
Sep 29 20:25:46 dist-test-slave-zbnc krb5kdc[31109](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713545, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.53.125@KRBTEST.COM for kudu/127.17.53.124@KRBTEST.COM
W20260929 20:25:46.384654 31291 tablet.cc:2369] T 00000000000000000000000000000000 P 38800a96109d41829eb37e95ff7286f7: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 20:25:46.387713 31143 tablet.cc:2369] T 00000000000000000000000000000000 P 5ddf8d4c8aee4b1ea7bcdc4c4f712f8f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 20:25:46.396704 31266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:25:46.375713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.124:44835 (local address 127.0.0.1:36792)
0929 20:25:46.375985 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0929 20:25:46.376007 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 20:25:46.376139 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:25:46.376535 (+ 396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:25:46.376541 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:25:46.377099 (+ 558us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:25:46.377345 (+ 246us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:25:46.377360 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:25:46.378930 (+ 1570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:25:46.378936 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:25:46.380361 (+ 1425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:25:46.380379 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:25:46.381403 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:25:46.381410 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:25:46.381647 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:25:46.382378 (+ 731us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:25:46.382406 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:25:46.385448 (+ 3042us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:25:46.393353 (+ 7905us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:25:46.393358 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:25:46.393363 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:25:46.393912 (+ 549us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:25:46.394363 (+ 451us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:25:46.394368 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:25:46.394372 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:25:46.394529 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:25:46.395453 (+ 924us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:25:46.395458 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:25:46.395633 (+ 175us) client_negotiation.cc:769] Sending connection context
0929 20:25:46.395908 (+ 275us) client_negotiation.cc:240] Negotiation successful
0929 20:25:46.396325 (+ 417us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147}
I20260929 20:25:46.397833 31340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:25:46.375795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36792 (local address 127.17.53.124:44835)
0929 20:25:46.375980 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0929 20:25:46.375989 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 20:25:46.376066 (+ 77us) server_negotiation.cc:407] Connection header received
0929 20:25:46.376284 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:25:46.376290 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:25:46.376369 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:25:46.376476 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:25:46.377510 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:25:46.378719 (+ 1209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:25:46.380522 (+ 1803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:25:46.381236 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:25:46.381361 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:25:46.385723 (+ 4362us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:25:46.385731 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:25:46.385737 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:25:46.385785 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:25:46.393171 (+ 7386us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:25:46.394065 (+ 894us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:25:46.394071 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:25:46.394074 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:25:46.394228 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:25:46.394683 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:25:46.394688 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:25:46.394692 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:25:46.395159 (+ 467us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:25:46.395386 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0929 20:25:46.397193 (+ 1807us) server_negotiation.cc:299] Negotiation successful
0929 20:25:46.397458 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 20:25:46.404819 31263 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6015ffd51e5d409cb73f3cbc6ee13ff9: Bootstrap starting.
I20260929 20:25:46.410677 31263 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6015ffd51e5d409cb73f3cbc6ee13ff9: Neither blocks nor log segments found. Creating new log.
I20260929 20:25:46.412036 31263 log.cc:824] T 00000000000000000000000000000000 P 6015ffd51e5d409cb73f3cbc6ee13ff9: Log is configured to *not* fsync() on all Append() calls
I20260929 20:25:46.415463 31263 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6015ffd51e5d409cb73f3cbc6ee13ff9: No bootstrap required, opened a new log
I20260929 20:25:46.428498 31263 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6015ffd51e5d409cb73f3cbc6ee13ff9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ddf8d4c8aee4b1ea7bcdc4c4f712f8f" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 44735 } } peers { permanent_uuid: "6015ffd51e5d409cb73f3cbc6ee13ff9" member_type: VOTER last_known_addr { host: "127.17.53.125" port: 37523 } } peers { permanent_uuid: "38800a96109d41829eb37e95ff7286f7" member_type: VOTER last_known_addr { host: "127.17.53.124" port: 44835 } }
I20260929 20:25:46.428961 31263 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6015ffd51e5d409cb73f3cbc6ee13ff9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:25:46.429077 31263 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6015ffd51e5d409cb73f3cbc6ee13ff9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6015ffd51e5d409cb73f3cbc6ee13ff9, State: Initialized, Role: FOLLOWER
I20260929 20:25:46.429781 31263 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6015ffd51e5d409cb73f3cbc6ee13ff9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ddf8d4c8aee4b1ea7bcdc4c4f712f8f" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 44735 } } peers { permanent_uuid: "6015ffd51e5d409cb73f3cbc6ee13ff9" member_type: VOTER last_known_addr { host: "127.17.53.125" port: 37523 } } peers { permanent_uuid: "38800a96109d41829eb37e95ff7286f7" member_type: VOTER last_known_addr { host: "127.17.53.124" port: 44835 } }
I20260929 20:25:46.430787 31378 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6015ffd51e5d409cb73f3cbc6ee13ff9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ddf8d4c8aee4b1ea7bcdc4c4f712f8f" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 44735 } } peers { permanent_uuid: "6015ffd51e5d409cb73f3cbc6ee13ff9" member_type: VOTER last_known_addr { host: "127.17.53.125" port: 37523 } } peers { permanent_uuid: "38800a96109d41829eb37e95ff7286f7" member_type: VOTER last_known_addr { host: "127.17.53.124" port: 44835 } } }
I20260929 20:25:46.431326 31378 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6015ffd51e5d409cb73f3cbc6ee13ff9 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 20:25:46.432199 31263 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6015ffd51e5d409cb73f3cbc6ee13ff9 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:25:46.453557 31196 master_runner.cc:263] No leader master found from master 6015ffd51e5d409cb73f3cbc6ee13ff9
I20260929 20:25:46.453722 31196 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260929 20:25:46.461704 31389 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6015ffd51e5d409cb73f3cbc6ee13ff9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 20:25:46.461992 31389 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 20:25:46.500932 31349 raft_consensus.cc:482] T 00000000000000000000000000000000 P 38800a96109d41829eb37e95ff7286f7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 20:25:46.501308 31349 raft_consensus.cc:504] T 00000000000000000000000000000000 P 38800a96109d41829eb37e95ff7286f7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ddf8d4c8aee4b1ea7bcdc4c4f712f8f" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 44735 } } peers { permanent_uuid: "6015ffd51e5d409cb73f3cbc6ee13ff9" member_type: VOTER last_known_addr { host: "127.17.53.125" port: 37523 } } peers { permanent_uuid: "38800a96109d41829eb37e95ff7286f7" member_type: VOTER last_known_addr { host: "127.17.53.124" port: 44835 } }
I20260929 20:25:46.503181 31349 leader_election.cc:288] T 00000000000000000000000000000000 P 38800a96109d41829eb37e95ff7286f7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5ddf8d4c8aee4b1ea7bcdc4c4f712f8f (127.17.53.126:44735), 6015ffd51e5d409cb73f3cbc6ee13ff9 (127.17.53.125:37523)
I20260929 20:25:46.504453 31238 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "38800a96109d41829eb37e95ff7286f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6015ffd51e5d409cb73f3cbc6ee13ff9" is_pre_election: true
I20260929 20:25:46.505136 31238 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 6015ffd51e5d409cb73f3cbc6ee13ff9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38800a96109d41829eb37e95ff7286f7 in term 0.
I20260929 20:25:46.506443 31164 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "38800a96109d41829eb37e95ff7286f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ddf8d4c8aee4b1ea7bcdc4c4f712f8f" is_pre_election: true
I20260929 20:25:46.507325 31286 leader_election.cc:302] T 00000000000000000000000000000000 P 38800a96109d41829eb37e95ff7286f7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38800a96109d41829eb37e95ff7286f7, 6015ffd51e5d409cb73f3cbc6ee13ff9; no voters:
I20260929 20:25:46.507933 31164 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 5ddf8d4c8aee4b1ea7bcdc4c4f712f8f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38800a96109d41829eb37e95ff7286f7 in term 0.
I20260929 20:25:46.513141 31349 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 38800a96109d41829eb37e95ff7286f7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 20:25:46.513304 31349 raft_consensus.cc:482] T 00000000000000000000000000000000 P 38800a96109d41829eb37e95ff7286f7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 20:25:46.513401 31349 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 38800a96109d41829eb37e95ff7286f7 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:25:46.517122 31349 raft_consensus.cc:504] T 00000000000000000000000000000000 P 38800a96109d41829eb37e95ff7286f7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ddf8d4c8aee4b1ea7bcdc4c4f712f8f" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 44735 } } peers { permanent_uuid: "6015ffd51e5d409cb73f3cbc6ee13ff9" member_type: VOTER last_known_addr { host: "127.17.53.125" port: 37523 } } peers { permanent_uuid: "38800a96109d41829eb37e95ff7286f7" member_type: VOTER last_known_addr { host: "127.17.53.124" port: 44835 } }
I20260929 20:25:46.518558 31164 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "38800a96109d41829eb37e95ff7286f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ddf8d4c8aee4b1ea7bcdc4c4f712f8f"
I20260929 20:25:46.518788 31164 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5ddf8d4c8aee4b1ea7bcdc4c4f712f8f [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:25:46.522344 31164 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 5ddf8d4c8aee4b1ea7bcdc4c4f712f8f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38800a96109d41829eb37e95ff7286f7 in term 1.
I20260929 20:25:46.522993 31287 leader_election.cc:302] T 00000000000000000000000000000000 P 38800a96109d41829eb37e95ff7286f7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38800a96109d41829eb37e95ff7286f7, 5ddf8d4c8aee4b1ea7bcdc4c4f712f8f; no voters:
I20260929 20:25:46.526120 31238 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "38800a96109d41829eb37e95ff7286f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6015ffd51e5d409cb73f3cbc6ee13ff9"
I20260929 20:25:46.526367 31238 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6015ffd51e5d409cb73f3cbc6ee13ff9 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:25:46.529798 31238 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 6015ffd51e5d409cb73f3cbc6ee13ff9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38800a96109d41829eb37e95ff7286f7 in term 1.
I20260929 20:25:46.530589 31349 leader_election.cc:288] T 00000000000000000000000000000000 P 38800a96109d41829eb37e95ff7286f7 [CANDIDATE]: Term 1 election: Requested vote from peers 5ddf8d4c8aee4b1ea7bcdc4c4f712f8f (127.17.53.126:44735), 6015ffd51e5d409cb73f3cbc6ee13ff9 (127.17.53.125:37523)
I20260929 20:25:46.531025 31349 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 38800a96109d41829eb37e95ff7286f7 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:25:46.534967 31349 raft_consensus.cc:686] T 00000000000000000000000000000000 P 38800a96109d41829eb37e95ff7286f7 [term 1 LEADER]: Becoming Leader. State: Replica: 38800a96109d41829eb37e95ff7286f7, State: Running, Role: LEADER
I20260929 20:25:46.536460 31349 consensus_queue.cc:237] T 00000000000000000000000000000000 P 38800a96109d41829eb37e95ff7286f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ddf8d4c8aee4b1ea7bcdc4c4f712f8f" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 44735 } } peers { permanent_uuid: "6015ffd51e5d409cb73f3cbc6ee13ff9" member_type: VOTER last_known_addr { host: "127.17.53.125" port: 37523 } } peers { permanent_uuid: "38800a96109d41829eb37e95ff7286f7" member_type: VOTER last_known_addr { host: "127.17.53.124" port: 44835 } }
I20260929 20:25:46.551733 31391 sys_catalog.cc:455] T 00000000000000000000000000000000 P 38800a96109d41829eb37e95ff7286f7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 38800a96109d41829eb37e95ff7286f7. Latest consensus state: current_term: 1 leader_uuid: "38800a96109d41829eb37e95ff7286f7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ddf8d4c8aee4b1ea7bcdc4c4f712f8f" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 44735 } } peers { permanent_uuid: "6015ffd51e5d409cb73f3cbc6ee13ff9" member_type: VOTER last_known_addr { host: "127.17.53.125" port: 37523 } } peers { permanent_uuid: "38800a96109d41829eb37e95ff7286f7" member_type: VOTER last_known_addr { host: "127.17.53.124" port: 44835 } } }
I20260929 20:25:46.552327 31391 sys_catalog.cc:458] T 00000000000000000000000000000000 P 38800a96109d41829eb37e95ff7286f7 [sys.catalog]: This master's current role is: LEADER
I20260929 20:25:46.559333 31393 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:25:46.561944 31393 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:25:46.583931 31238 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 6015ffd51e5d409cb73f3cbc6ee13ff9 [term 1 FOLLOWER]: Refusing update from remote peer 38800a96109d41829eb37e95ff7286f7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:25:46.584996 31391 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 38800a96109d41829eb37e95ff7286f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6015ffd51e5d409cb73f3cbc6ee13ff9" member_type: VOTER last_known_addr { host: "127.17.53.125" port: 37523 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:25:46.592948 31164 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 5ddf8d4c8aee4b1ea7bcdc4c4f712f8f [term 1 FOLLOWER]: Refusing update from remote peer 38800a96109d41829eb37e95ff7286f7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:25:46.593901 31391 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 38800a96109d41829eb37e95ff7286f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ddf8d4c8aee4b1ea7bcdc4c4f712f8f" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 44735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:25:46.622748 31364 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5ddf8d4c8aee4b1ea7bcdc4c4f712f8f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 38800a96109d41829eb37e95ff7286f7. Latest consensus state: current_term: 1 leader_uuid: "38800a96109d41829eb37e95ff7286f7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ddf8d4c8aee4b1ea7bcdc4c4f712f8f" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 44735 } } peers { permanent_uuid: "6015ffd51e5d409cb73f3cbc6ee13ff9" member_type: VOTER last_known_addr { host: "127.17.53.125" port: 37523 } } peers { permanent_uuid: "38800a96109d41829eb37e95ff7286f7" member_type: VOTER last_known_addr { host: "127.17.53.124" port: 44835 } } }
I20260929 20:25:46.623106 31364 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5ddf8d4c8aee4b1ea7bcdc4c4f712f8f [sys.catalog]: This master's current role is: FOLLOWER
I20260929 20:25:46.625946 31394 mvcc.cc:205] Tried to move back new op lower bound from 7334762686794895360 to 7334762686625005568. Current Snapshot: MvccSnapshot[applied={T|T < 7334762686794895360}]
I20260929 20:25:46.638972 31364 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5ddf8d4c8aee4b1ea7bcdc4c4f712f8f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "38800a96109d41829eb37e95ff7286f7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ddf8d4c8aee4b1ea7bcdc4c4f712f8f" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 44735 } } peers { permanent_uuid: "6015ffd51e5d409cb73f3cbc6ee13ff9" member_type: VOTER last_known_addr { host: "127.17.53.125" port: 37523 } } peers { permanent_uuid: "38800a96109d41829eb37e95ff7286f7" member_type: VOTER last_known_addr { host: "127.17.53.124" port: 44835 } } }
I20260929 20:25:46.639281 31364 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5ddf8d4c8aee4b1ea7bcdc4c4f712f8f [sys.catalog]: This master's current role is: FOLLOWER
I20260929 20:25:46.639679 31349 sys_catalog.cc:455] T 00000000000000000000000000000000 P 38800a96109d41829eb37e95ff7286f7 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "38800a96109d41829eb37e95ff7286f7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ddf8d4c8aee4b1ea7bcdc4c4f712f8f" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 44735 } } peers { permanent_uuid: "6015ffd51e5d409cb73f3cbc6ee13ff9" member_type: VOTER last_known_addr { host: "127.17.53.125" port: 37523 } } peers { permanent_uuid: "38800a96109d41829eb37e95ff7286f7" member_type: VOTER last_known_addr { host: "127.17.53.124" port: 44835 } } }
I20260929 20:25:46.639982 31349 sys_catalog.cc:458] T 00000000000000000000000000000000 P 38800a96109d41829eb37e95ff7286f7 [sys.catalog]: This master's current role is: LEADER
W20260929 20:25:46.639952 31217 tablet.cc:2369] T 00000000000000000000000000000000 P 6015ffd51e5d409cb73f3cbc6ee13ff9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 20:25:46.641709 31393 catalog_manager.cc:1347] Generated new cluster ID: ff166c3f06a6454b947430573bc85543
I20260929 20:25:46.641835 31393 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:25:46.648313 31391 sys_catalog.cc:455] T 00000000000000000000000000000000 P 38800a96109d41829eb37e95ff7286f7 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "38800a96109d41829eb37e95ff7286f7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ddf8d4c8aee4b1ea7bcdc4c4f712f8f" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 44735 } } peers { permanent_uuid: "6015ffd51e5d409cb73f3cbc6ee13ff9" member_type: VOTER last_known_addr { host: "127.17.53.125" port: 37523 } } peers { permanent_uuid: "38800a96109d41829eb37e95ff7286f7" member_type: VOTER last_known_addr { host: "127.17.53.124" port: 44835 } } }
I20260929 20:25:46.648558 31391 sys_catalog.cc:458] T 00000000000000000000000000000000 P 38800a96109d41829eb37e95ff7286f7 [sys.catalog]: This master's current role is: LEADER
I20260929 20:25:46.648588 31378 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6015ffd51e5d409cb73f3cbc6ee13ff9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 38800a96109d41829eb37e95ff7286f7. Latest consensus state: current_term: 1 leader_uuid: "38800a96109d41829eb37e95ff7286f7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ddf8d4c8aee4b1ea7bcdc4c4f712f8f" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 44735 } } peers { permanent_uuid: "6015ffd51e5d409cb73f3cbc6ee13ff9" member_type: VOTER last_known_addr { host: "127.17.53.125" port: 37523 } } peers { permanent_uuid: "38800a96109d41829eb37e95ff7286f7" member_type: VOTER last_known_addr { host: "127.17.53.124" port: 44835 } } }
I20260929 20:25:46.648891 31378 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6015ffd51e5d409cb73f3cbc6ee13ff9 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 20:25:46.686599 31378 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6015ffd51e5d409cb73f3cbc6ee13ff9 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "38800a96109d41829eb37e95ff7286f7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5ddf8d4c8aee4b1ea7bcdc4c4f712f8f" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 44735 } } peers { permanent_uuid: "6015ffd51e5d409cb73f3cbc6ee13ff9" member_type: VOTER last_known_addr { host: "127.17.53.125" port: 37523 } } peers { permanent_uuid: "38800a96109d41829eb37e95ff7286f7" member_type: VOTER last_known_addr { host: "127.17.53.124" port: 44835 } } }
I20260929 20:25:46.686882 31378 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6015ffd51e5d409cb73f3cbc6ee13ff9 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 20:25:46.716383 31393 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 20:25:46.719259 31393 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:25:46.742983 31393 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 38800a96109d41829eb37e95ff7286f7: Generated new TSK 0
I20260929 20:25:46.744154 31393 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 20:25:47.347097 31375 catalog_manager.cc:1259] Loaded cluster ID: ff166c3f06a6454b947430573bc85543
I20260929 20:25:47.347196 31375 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 5ddf8d4c8aee4b1ea7bcdc4c4f712f8f: loading cluster ID for follower catalog manager: success
I20260929 20:25:47.349776 31375 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 5ddf8d4c8aee4b1ea7bcdc4c4f712f8f: acquiring CA information for follower catalog manager: success
I20260929 20:25:47.351627 31375 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 5ddf8d4c8aee4b1ea7bcdc4c4f712f8f: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 20:25:47.464242 31389 catalog_manager.cc:1259] Loaded cluster ID: ff166c3f06a6454b947430573bc85543
I20260929 20:25:47.464346 31389 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 6015ffd51e5d409cb73f3cbc6ee13ff9: loading cluster ID for follower catalog manager: success
I20260929 20:25:47.466845 31389 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 6015ffd51e5d409cb73f3cbc6ee13ff9: acquiring CA information for follower catalog manager: success
I20260929 20:25:47.468688 31389 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 6015ffd51e5d409cb73f3cbc6ee13ff9: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 20:25:47.903147 17621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu pid 31122
I20260929 20:25:47.904277 31184 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:25:48.013509 17621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu with pid 31122
I20260929 20:25:48.034502 17621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu pid 31196
I20260929 20:25:48.035800 31258 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:25:48.064888 31409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:25:48.064100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:44735 (local address 127.0.0.1:50576)
0929 20:25:48.064607 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0929 20:25:48.064717 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.53.126:44735: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":297,"threads_started":1}
W20260929 20:25:48.065200 31287 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.53.126:44735: connect: Connection refused (error 111)
I20260929 20:25:48.067956 31409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:25:48.067596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:44735 (local address 127.0.0.1:50586)
0929 20:25:48.067759 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0929 20:25:48.067840 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.53.126:44735: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260929 20:25:48.068147 31287 consensus_peers.cc:487] T 00000000000000000000000000000000 P 38800a96109d41829eb37e95ff7286f7 -> Peer 5ddf8d4c8aee4b1ea7bcdc4c4f712f8f (127.17.53.126:44735): Couldn't send request to peer 5ddf8d4c8aee4b1ea7bcdc4c4f712f8f. Status: Network error: Client connection negotiation failed: client connection to 127.17.53.126:44735: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260929 20:25:48.143189 17621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu with pid 31196
I20260929 20:25:48.164232 17621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu pid 31270
I20260929 20:25:48.165562 31332 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:25:48.276978 17621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu with pid 31270
I20260929 20:25:48.277091 31409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:25:48.276445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.125:37523 (local address 127.0.0.1:55390)
0929 20:25:48.276753 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0929 20:25:48.276882 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.53.125:37523: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124}
2026-09-29T20:25:48Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (4898 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790713339161492-17621-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 20:25:48 dist-test-slave-zbnc krb5kdc[31414](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 20:25:48 dist-test-slave-zbnc krb5kdc[31414](info): set up 2 sockets
Sep 29 20:25:48 dist-test-slave-zbnc krb5kdc[31414](info): commencing operation
krb5kdc: starting...
W20260929 20:25:49.380823 17621 mini_kdc.cc:121] Time spent starting KDC: real 1.059s user 0.004s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790713339161492-17621-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790713339161492-17621-0/krb5kdc/test-user.keytab.
Sep 29 20:25:49 dist-test-slave-zbnc krb5kdc[31414](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713549, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T20:25:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T20:25:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.17.53.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.53.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.53.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.126.keytab.
Entry for principal kudu/127.17.53.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.126.keytab.
I20260929 20:25:49.618901 17621 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
/tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790713339161492-17621-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790713339161492-17621-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790713339161492-17621-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790713339161492-17621-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.53.126:38245
--webserver_interface=127.17.53.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.126.keytab
--principal=kudu/127.17.53.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.53.84:40047
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.53.126:38245
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790713339161492-17621-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790713339161492-17621-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790713339161492-17621-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:25:49.953125 31427 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:25:49.953436 31427 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:25:49.953528 31427 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:25:49.961225 31427 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:25:49.961349 31427 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:25:49.961396 31427 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:25:49.961455 31427 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:25:49.961499 31427 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:25:49.971557 31427 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.53.84:40047
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790713339161492-17621-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790713339161492-17621-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.53.126:38245
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.126.keytab
--principal=kudu/127.17.53.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.53.126:38245
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790713339161492-17621-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.53.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790713339161492-17621-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:25:49.972900 31427 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:25:49.974151 31427 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:25:49.986500 31433 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:25:49.986545 31432 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:25:49.986446 31435 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:25:49.992395 31427 server_base.cc:1034] running on GCE node
I20260929 20:25:49.993230 31427 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:25:49.994706 31427 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:25:49.995950 31427 hybrid_clock.cc:648] HybridClock initialized: now 1790713549995903 us; error 63 us; skew 500 ppm
I20260929 20:25:49.996318 31427 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:25:50 dist-test-slave-zbnc krb5kdc[31414](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713550, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.53.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:25:50.005275 31427 init.cc:377] Logged in from keytab as kudu/127.17.53.126@KRBTEST.COM (short username kudu)
I20260929 20:25:50.006968 31427 webserver.cc:460] Webserver started at http://127.17.53.126:46593/ using document root <none> and password file <none>
I20260929 20:25:50.007742 31427 fs_manager.cc:359] Metadata directory not provided
I20260929 20:25:50.007895 31427 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:25:50.008221 31427 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:25:50.011415 31427 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790713339161492-17621-0/minicluster-data/master-0/data/instance:
uuid: "849e1a01f1194262a6ba0d819f5f93c5"
format_stamp: "Formatted at 2026-09-29 20:25:50 on dist-test-slave-zbnc"
server_key: "04d6aa2c1a003690ac2ca10fc372d89c"
server_key_iv: "5d75f88b3e971358e0e1d137542399e3"
server_key_version: "encryptionkey@0"
I20260929 20:25:50.012185 31427 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790713339161492-17621-0/minicluster-data/master-0/wal/instance:
uuid: "849e1a01f1194262a6ba0d819f5f93c5"
format_stamp: "Formatted at 2026-09-29 20:25:50 on dist-test-slave-zbnc"
server_key: "04d6aa2c1a003690ac2ca10fc372d89c"
server_key_iv: "5d75f88b3e971358e0e1d137542399e3"
server_key_version: "encryptionkey@0"
I20260929 20:25:50.017508 31427 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 20:25:50.021037 31442 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:25:50.022334 31427 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 20:25:50.022512 31427 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790713339161492-17621-0/minicluster-data/master-0/data,/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790713339161492-17621-0/minicluster-data/master-0/wal
uuid: "849e1a01f1194262a6ba0d819f5f93c5"
format_stamp: "Formatted at 2026-09-29 20:25:50 on dist-test-slave-zbnc"
server_key: "04d6aa2c1a003690ac2ca10fc372d89c"
server_key_iv: "5d75f88b3e971358e0e1d137542399e3"
server_key_version: "encryptionkey@0"
I20260929 20:25:50.022709 31427 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790713339161492-17621-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790713339161492-17621-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790713339161492-17621-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:25:50.043347 31427 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:25:50.047489 31427 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:25:50.047780 31427 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:25:50.066313 31427 rpc_server.cc:307] RPC server started. Bound to: 127.17.53.126:38245
I20260929 20:25:50.066344 31493 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.53.126:38245 every 8 connection(s)
I20260929 20:25:50.067765 31427 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790713339161492-17621-0/minicluster-data/master-0/data/info.pb
I20260929 20:25:50.071635 31494 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:25:50.072397 17621 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu as pid 31427
I20260929 20:25:50.072620 17621 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790713339161492-17621-0/minicluster-data/master-0/wal/instance
I20260929 20:25:50.073311 17621 external_mini_cluster.cc:1428] Setting key 2efc8006302a1cba86068b25e958f2b6
I20260929 20:25:50.086289 31494 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 849e1a01f1194262a6ba0d819f5f93c5: Bootstrap starting.
Sep 29 20:25:50 dist-test-slave-zbnc krb5kdc[31414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713549, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.53.126@KRBTEST.COM
I20260929 20:25:50.089646 31494 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 849e1a01f1194262a6ba0d819f5f93c5: Neither blocks nor log segments found. Creating new log.
I20260929 20:25:50.090813 31494 log.cc:824] T 00000000000000000000000000000000 P 849e1a01f1194262a6ba0d819f5f93c5: Log is configured to *not* fsync() on all Append() calls
I20260929 20:25:50.094090 31494 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 849e1a01f1194262a6ba0d819f5f93c5: No bootstrap required, opened a new log
I20260929 20:25:50.097731 31497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:25:50.075243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48164 (local address 127.17.53.126:38245)
0929 20:25:50.076123 (+ 880us) server_negotiation.cc:206] Beginning negotiation
0929 20:25:50.076140 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0929 20:25:50.076200 (+ 60us) server_negotiation.cc:407] Connection header received
0929 20:25:50.078448 (+ 2248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:25:50.078477 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:25:50.079051 (+ 574us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:25:50.079530 (+ 479us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:25:50.080840 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:25:50.082898 (+ 2058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:25:50.084231 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:25:50.085282 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:25:50.085462 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:25:50.090016 (+ 4554us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:25:50.090026 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:25:50.090041 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:25:50.090080 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:25:50.093261 (+ 3181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:25:50.094016 (+ 755us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:25:50.094029 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:25:50.094034 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:25:50.094288 (+ 254us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:25:50.094685 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:25:50.094690 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:25:50.094694 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:25:50.095340 (+ 646us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:25:50.095657 (+ 317us) server_negotiation.cc:1035] Waiting for connection context
0929 20:25:50.096183 (+ 526us) server_negotiation.cc:299] Negotiation successful
0929 20:25:50.096661 (+ 478us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":618,"thread_start_us":395,"threads_started":1}
I20260929 20:25:50.102084 17621 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 20:25:50.103329 31494 raft_consensus.cc:348] T 00000000000000000000000000000000 P 849e1a01f1194262a6ba0d819f5f93c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "849e1a01f1194262a6ba0d819f5f93c5" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 38245 } }
I20260929 20:25:50.103735 31494 raft_consensus.cc:374] T 00000000000000000000000000000000 P 849e1a01f1194262a6ba0d819f5f93c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:25:50.103849 31494 raft_consensus.cc:729] T 00000000000000000000000000000000 P 849e1a01f1194262a6ba0d819f5f93c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 849e1a01f1194262a6ba0d819f5f93c5, State: Initialized, Role: FOLLOWER
I20260929 20:25:50.104805 31494 consensus_queue.cc:260] T 00000000000000000000000000000000 P 849e1a01f1194262a6ba0d819f5f93c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "849e1a01f1194262a6ba0d819f5f93c5" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 38245 } }
I20260929 20:25:50.105123 31494 raft_consensus.cc:388] T 00000000000000000000000000000000 P 849e1a01f1194262a6ba0d819f5f93c5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 20:25:50.105279 31494 raft_consensus.cc:482] T 00000000000000000000000000000000 P 849e1a01f1194262a6ba0d819f5f93c5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 20:25:50.105542 31494 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 849e1a01f1194262a6ba0d819f5f93c5 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:25:50.107839 31494 raft_consensus.cc:504] T 00000000000000000000000000000000 P 849e1a01f1194262a6ba0d819f5f93c5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "849e1a01f1194262a6ba0d819f5f93c5" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 38245 } }
I20260929 20:25:50.108330 31494 leader_election.cc:302] T 00000000000000000000000000000000 P 849e1a01f1194262a6ba0d819f5f93c5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 849e1a01f1194262a6ba0d819f5f93c5; no voters:
I20260929 20:25:50.108860 31494 leader_election.cc:288] T 00000000000000000000000000000000 P 849e1a01f1194262a6ba0d819f5f93c5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 20:25:50.109014 31499 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 849e1a01f1194262a6ba0d819f5f93c5 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:25:50.109730 31499 raft_consensus.cc:686] T 00000000000000000000000000000000 P 849e1a01f1194262a6ba0d819f5f93c5 [term 1 LEADER]: Becoming Leader. State: Replica: 849e1a01f1194262a6ba0d819f5f93c5, State: Running, Role: LEADER
I20260929 20:25:50.110330 31499 consensus_queue.cc:237] T 00000000000000000000000000000000 P 849e1a01f1194262a6ba0d819f5f93c5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "849e1a01f1194262a6ba0d819f5f93c5" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 38245 } }
I20260929 20:25:50.111078 31494 sys_catalog.cc:564] T 00000000000000000000000000000000 P 849e1a01f1194262a6ba0d819f5f93c5 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:25:50.113360 31500 sys_catalog.cc:455] T 00000000000000000000000000000000 P 849e1a01f1194262a6ba0d819f5f93c5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "849e1a01f1194262a6ba0d819f5f93c5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "849e1a01f1194262a6ba0d819f5f93c5" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 38245 } } }
I20260929 20:25:50.113823 31500 sys_catalog.cc:458] T 00000000000000000000000000000000 P 849e1a01f1194262a6ba0d819f5f93c5 [sys.catalog]: This master's current role is: LEADER
I20260929 20:25:50.114631 31501 sys_catalog.cc:455] T 00000000000000000000000000000000 P 849e1a01f1194262a6ba0d819f5f93c5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 849e1a01f1194262a6ba0d819f5f93c5. Latest consensus state: current_term: 1 leader_uuid: "849e1a01f1194262a6ba0d819f5f93c5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "849e1a01f1194262a6ba0d819f5f93c5" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 38245 } } }
I20260929 20:25:50.115504 31501 sys_catalog.cc:458] T 00000000000000000000000000000000 P 849e1a01f1194262a6ba0d819f5f93c5 [sys.catalog]: This master's current role is: LEADER
I20260929 20:25:50.115789 31506 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:25:50.121446 31506 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:25:50.130041 31506 catalog_manager.cc:1347] Generated new cluster ID: 746db14f39fc42debfde3a871bb0b1a2
I20260929 20:25:50.130172 31506 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:25:50.144980 31506 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 20:25:50.146538 31506 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:25:50.162074 31506 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 849e1a01f1194262a6ba0d819f5f93c5: Generated new TSK 0
I20260929 20:25:50.163074 31506 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 20:25:50.215066 17621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu pid 31427
I20260929 20:25:50.216301 31489 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:25:50.315892 17621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu with pid 31427
2026-09-29T20:25:50Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (2032 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713339161492-17621-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 20:25:50 dist-test-slave-zbnc krb5kdc[31522](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 20:25:50 dist-test-slave-zbnc krb5kdc[31522](info): set up 2 sockets
Sep 29 20:25:50 dist-test-slave-zbnc krb5kdc[31522](info): commencing operation
krb5kdc: starting...
W20260929 20:25:51.468784 17621 mini_kdc.cc:121] Time spent starting KDC: real 1.114s user 0.000s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713339161492-17621-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713339161492-17621-0/krb5kdc/test-user.keytab.
Sep 29 20:25:51 dist-test-slave-zbnc krb5kdc[31522](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713551, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T20:25:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T20:25:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.17.53.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.53.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.53.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.126.keytab.
Entry for principal kudu/127.17.53.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.126.keytab.
I20260929 20:25:51.710541 17621 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
/tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713339161492-17621-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713339161492-17621-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713339161492-17621-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713339161492-17621-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.53.126:37213
--webserver_interface=127.17.53.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.126.keytab
--principal=kudu/127.17.53.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.53.84:41567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.53.126:37213
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713339161492-17621-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713339161492-17621-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713339161492-17621-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:25:52.043366 31535 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:25:52.043716 31535 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:25:52.043885 31535 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:25:52.051252 31535 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:25:52.051395 31535 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:25:52.051494 31535 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:25:52.051596 31535 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:25:52.051674 31535 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:25:52.061985 31535 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.53.84:41567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713339161492-17621-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713339161492-17621-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.53.126:37213
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.126.keytab
--principal=kudu/127.17.53.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.53.126:37213
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713339161492-17621-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.53.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713339161492-17621-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:25:52.063402 31535 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:25:52.064723 31535 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:25:52.077692 31540 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:25:52.077780 31543 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:25:52.077809 31541 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:25:52.078927 31535 server_base.cc:1034] running on GCE node
I20260929 20:25:52.079686 31535 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:25:52.081171 31535 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:25:52.082393 31535 hybrid_clock.cc:648] HybridClock initialized: now 1790713552082335 us; error 50 us; skew 500 ppm
I20260929 20:25:52.082751 31535 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:25:52 dist-test-slave-zbnc krb5kdc[31522](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713552, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.53.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:25:52.085922 31535 init.cc:377] Logged in from keytab as kudu/127.17.53.126@KRBTEST.COM (short username kudu)
I20260929 20:25:52.087498 31535 webserver.cc:460] Webserver started at http://127.17.53.126:40845/ using document root <none> and password file <none>
I20260929 20:25:52.088205 31535 fs_manager.cc:359] Metadata directory not provided
I20260929 20:25:52.088317 31535 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:25:52.088621 31535 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:25:52.091787 31535 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713339161492-17621-0/minicluster-data/master-0/data/instance:
uuid: "7caaae55b00c4da99832e09118f54e5e"
format_stamp: "Formatted at 2026-09-29 20:25:52 on dist-test-slave-zbnc"
server_key: "556bb7331b6e6cd015f30def5b023699"
server_key_iv: "e08504e502dda6de1daf7acf50e383a6"
server_key_version: "encryptionkey@0"
I20260929 20:25:52.092558 31535 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713339161492-17621-0/minicluster-data/master-0/wal/instance:
uuid: "7caaae55b00c4da99832e09118f54e5e"
format_stamp: "Formatted at 2026-09-29 20:25:52 on dist-test-slave-zbnc"
server_key: "556bb7331b6e6cd015f30def5b023699"
server_key_iv: "e08504e502dda6de1daf7acf50e383a6"
server_key_version: "encryptionkey@0"
I20260929 20:25:52.097896 31535 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260929 20:25:52.101560 31550 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:25:52.103036 31535 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 20:25:52.103223 31535 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713339161492-17621-0/minicluster-data/master-0/data,/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713339161492-17621-0/minicluster-data/master-0/wal
uuid: "7caaae55b00c4da99832e09118f54e5e"
format_stamp: "Formatted at 2026-09-29 20:25:52 on dist-test-slave-zbnc"
server_key: "556bb7331b6e6cd015f30def5b023699"
server_key_iv: "e08504e502dda6de1daf7acf50e383a6"
server_key_version: "encryptionkey@0"
I20260929 20:25:52.103447 31535 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713339161492-17621-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713339161492-17621-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713339161492-17621-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:25:52.125365 31535 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:25:52.131242 31535 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:25:52.131600 31535 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:25:52.154073 31601 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.53.126:37213 every 8 connection(s)
I20260929 20:25:52.154067 31535 rpc_server.cc:307] RPC server started. Bound to: 127.17.53.126:37213
I20260929 20:25:52.156365 31535 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713339161492-17621-0/minicluster-data/master-0/data/info.pb
I20260929 20:25:52.160897 31602 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:25:52.163030 17621 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu as pid 31535
I20260929 20:25:52.163259 17621 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713339161492-17621-0/minicluster-data/master-0/wal/instance
I20260929 20:25:52.164117 17621 external_mini_cluster.cc:1428] Setting key 7f419d19314446fa3fd927c571281cb3
I20260929 20:25:52.175305 31602 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7caaae55b00c4da99832e09118f54e5e: Bootstrap starting.
I20260929 20:25:52.179201 31602 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7caaae55b00c4da99832e09118f54e5e: Neither blocks nor log segments found. Creating new log.
I20260929 20:25:52.180327 31602 log.cc:824] T 00000000000000000000000000000000 P 7caaae55b00c4da99832e09118f54e5e: Log is configured to *not* fsync() on all Append() calls
I20260929 20:25:52.183646 31602 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7caaae55b00c4da99832e09118f54e5e: No bootstrap required, opened a new log
Sep 29 20:25:52 dist-test-slave-zbnc krb5kdc[31522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713551, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.53.126@KRBTEST.COM
I20260929 20:25:52.193542 31602 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7caaae55b00c4da99832e09118f54e5e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7caaae55b00c4da99832e09118f54e5e" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 37213 } }
I20260929 20:25:52.194061 31602 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7caaae55b00c4da99832e09118f54e5e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:25:52.194183 31602 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7caaae55b00c4da99832e09118f54e5e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7caaae55b00c4da99832e09118f54e5e, State: Initialized, Role: FOLLOWER
I20260929 20:25:52.194952 31602 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7caaae55b00c4da99832e09118f54e5e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7caaae55b00c4da99832e09118f54e5e" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 37213 } }
I20260929 20:25:52.195209 31602 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7caaae55b00c4da99832e09118f54e5e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 20:25:52.195367 31602 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7caaae55b00c4da99832e09118f54e5e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 20:25:52.195559 31602 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7caaae55b00c4da99832e09118f54e5e [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:25:52.198019 31602 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7caaae55b00c4da99832e09118f54e5e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7caaae55b00c4da99832e09118f54e5e" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 37213 } }
I20260929 20:25:52.198104 31605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:25:52.166903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53008 (local address 127.17.53.126:37213)
0929 20:25:52.174121 (+ 7218us) server_negotiation.cc:206] Beginning negotiation
0929 20:25:52.174135 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 20:25:52.174184 (+ 49us) server_negotiation.cc:407] Connection header received
0929 20:25:52.176180 (+ 1996us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:25:52.176214 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:25:52.176616 (+ 402us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:25:52.176995 (+ 379us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:25:52.178999 (+ 2004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:25:52.180492 (+ 1493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:25:52.182278 (+ 1786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:25:52.183068 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:25:52.183213 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:25:52.189850 (+ 6637us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:25:52.189861 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:25:52.189878 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:25:52.189927 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:25:52.193313 (+ 3386us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:25:52.194247 (+ 934us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:25:52.194262 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:25:52.194271 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:25:52.194480 (+ 209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:25:52.194945 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:25:52.194950 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:25:52.194954 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:25:52.195736 (+ 782us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:25:52.196066 (+ 330us) server_negotiation.cc:1035] Waiting for connection context
0929 20:25:52.196586 (+ 520us) server_negotiation.cc:299] Negotiation successful
0929 20:25:52.197055 (+ 469us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6960,"thread_start_us":6703,"threads_started":1}
I20260929 20:25:52.198576 31602 leader_election.cc:302] T 00000000000000000000000000000000 P 7caaae55b00c4da99832e09118f54e5e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7caaae55b00c4da99832e09118f54e5e; no voters:
I20260929 20:25:52.199345 31602 leader_election.cc:288] T 00000000000000000000000000000000 P 7caaae55b00c4da99832e09118f54e5e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 20:25:52.199483 31607 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7caaae55b00c4da99832e09118f54e5e [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:25:52.202056 31602 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7caaae55b00c4da99832e09118f54e5e [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:25:52.203125 31607 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7caaae55b00c4da99832e09118f54e5e [term 1 LEADER]: Becoming Leader. State: Replica: 7caaae55b00c4da99832e09118f54e5e, State: Running, Role: LEADER
I20260929 20:25:52.203796 31607 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7caaae55b00c4da99832e09118f54e5e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7caaae55b00c4da99832e09118f54e5e" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 37213 } }
I20260929 20:25:52.207000 31609 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7caaae55b00c4da99832e09118f54e5e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7caaae55b00c4da99832e09118f54e5e. Latest consensus state: current_term: 1 leader_uuid: "7caaae55b00c4da99832e09118f54e5e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7caaae55b00c4da99832e09118f54e5e" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 37213 } } }
I20260929 20:25:52.207086 31608 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7caaae55b00c4da99832e09118f54e5e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7caaae55b00c4da99832e09118f54e5e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7caaae55b00c4da99832e09118f54e5e" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 37213 } } }
I20260929 20:25:52.207381 31609 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7caaae55b00c4da99832e09118f54e5e [sys.catalog]: This master's current role is: LEADER
I20260929 20:25:52.208806 31608 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7caaae55b00c4da99832e09118f54e5e [sys.catalog]: This master's current role is: LEADER
I20260929 20:25:52.209702 31617 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:25:52.215785 31617 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:25:52.225244 31617 catalog_manager.cc:1347] Generated new cluster ID: 4893bb9645e249e4b64ef32dfa3ad00f
I20260929 20:25:52.225373 31617 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:25:52.248517 31617 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 20:25:52.250545 31617 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:25:52.270776 31617 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7caaae55b00c4da99832e09118f54e5e: Generated new TSK 0
I20260929 20:25:52.271989 31617 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.17.53.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.53.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.53.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.65.keytab.
Entry for principal kudu/127.17.53.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.65.keytab.
I20260929 20:25:52.343947 17621 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
/tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713339161492-17621-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713339161492-17621-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713339161492-17621-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713339161492-17621-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.53.65:0
--local_ip_for_outbound_sockets=127.17.53.65
--webserver_interface=127.17.53.65
--webserver_port=0
--tserver_master_addrs=127.17.53.126:37213
--keytab_file=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.65.keytab
--principal=kudu/127.17.53.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.53.84:41567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713339161492-17621-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713339161492-17621-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713339161492-17621-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:25:52.681960 31628 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:25:52.682302 31628 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:25:52.682461 31628 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:25:52.689894 31628 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:25:52.690059 31628 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:25:52.690239 31628 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.53.65
I20260929 20:25:52.700153 31628 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.53.84:41567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713339161492-17621-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713339161492-17621-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.65.keytab
--principal=kudu/127.17.53.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.53.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713339161492-17621-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.17.53.65
--webserver_port=0
--tserver_master_addrs=127.17.53.126:37213
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.53.65
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713339161492-17621-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:25:52.701558 31628 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:25:52.703141 31628 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:25:52.717173 31634 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:25:52.717216 31636 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:25:52.717798 31633 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:25:52.718166 31628 server_base.cc:1034] running on GCE node
I20260929 20:25:52.718910 31628 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:25:52.719907 31628 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:25:52.721130 31628 hybrid_clock.cc:648] HybridClock initialized: now 1790713552721051 us; error 78 us; skew 500 ppm
I20260929 20:25:52.721452 31628 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:25:52 dist-test-slave-zbnc krb5kdc[31522](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713552, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.53.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:25:52.724624 31628 init.cc:377] Logged in from keytab as kudu/127.17.53.65@KRBTEST.COM (short username kudu)
I20260929 20:25:52.726063 31628 webserver.cc:460] Webserver started at http://127.17.53.65:46229/ using document root <none> and password file <none>
I20260929 20:25:52.726768 31628 fs_manager.cc:359] Metadata directory not provided
I20260929 20:25:52.726943 31628 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:25:52.727255 31628 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:25:52.730250 31628 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713339161492-17621-0/minicluster-data/ts-0/data/instance:
uuid: "ad1b6c08928b446ca0b8539b56223f57"
format_stamp: "Formatted at 2026-09-29 20:25:52 on dist-test-slave-zbnc"
server_key: "ea98d3192c7fc95189919bf5c74d03ae"
server_key_iv: "03dc1c8d9834e63fb29a049126bdefe3"
server_key_version: "encryptionkey@0"
I20260929 20:25:52.731108 31628 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713339161492-17621-0/minicluster-data/ts-0/wal/instance:
uuid: "ad1b6c08928b446ca0b8539b56223f57"
format_stamp: "Formatted at 2026-09-29 20:25:52 on dist-test-slave-zbnc"
server_key: "ea98d3192c7fc95189919bf5c74d03ae"
server_key_iv: "03dc1c8d9834e63fb29a049126bdefe3"
server_key_version: "encryptionkey@0"
I20260929 20:25:52.736353 31628 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260929 20:25:52.739883 31643 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:25:52.741387 31628 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 20:25:52.741565 31628 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713339161492-17621-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713339161492-17621-0/minicluster-data/ts-0/wal
uuid: "ad1b6c08928b446ca0b8539b56223f57"
format_stamp: "Formatted at 2026-09-29 20:25:52 on dist-test-slave-zbnc"
server_key: "ea98d3192c7fc95189919bf5c74d03ae"
server_key_iv: "03dc1c8d9834e63fb29a049126bdefe3"
server_key_version: "encryptionkey@0"
I20260929 20:25:52.741791 31628 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713339161492-17621-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713339161492-17621-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713339161492-17621-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:25:52.764653 31628 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:25:52.768812 31628 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:25:52.769115 31628 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:25:52.770185 31628 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:25:52.771818 31628 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:25:52.771893 31628 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:25:52.771981 31628 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:25:52.772025 31628 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:25:52.806039 31628 rpc_server.cc:307] RPC server started. Bound to: 127.17.53.65:36129
I20260929 20:25:52.806099 31755 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.53.65:36129 every 8 connection(s)
I20260929 20:25:52.807497 31628 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713339161492-17621-0/minicluster-data/ts-0/data/info.pb
I20260929 20:25:52.809865 17621 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu as pid 31628
I20260929 20:25:52.810111 17621 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713339161492-17621-0/minicluster-data/ts-0/wal/instance
I20260929 20:25:52.810820 17621 external_mini_cluster.cc:1428] Setting key c0b2f9330655e37ba3bbb1dfed672984
Sep 29 20:25:52 dist-test-slave-zbnc krb5kdc[31522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713552, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.53.65@KRBTEST.COM for kudu/127.17.53.126@KRBTEST.COM
I20260929 20:25:52.830025 31758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:25:52.810157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.53.65:57231 (local address 127.17.53.126:37213)
0929 20:25:52.812991 (+ 2834us) server_negotiation.cc:206] Beginning negotiation
0929 20:25:52.812996 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:25:52.813014 (+ 18us) server_negotiation.cc:407] Connection header received
0929 20:25:52.814081 (+ 1067us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:25:52.814088 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:25:52.814168 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:25:52.814286 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:25:52.815877 (+ 1591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:25:52.817045 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:25:52.818588 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:25:52.819337 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:25:52.819479 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:25:52.822943 (+ 3464us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:25:52.822953 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:25:52.822961 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:25:52.823005 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:25:52.826549 (+ 3544us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:25:52.827378 (+ 829us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:25:52.827386 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:25:52.827392 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:25:52.827543 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:25:52.827987 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:25:52.827994 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:25:52.828001 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:25:52.828405 (+ 404us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:25:52.828558 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0929 20:25:52.829295 (+ 737us) server_negotiation.cc:299] Negotiation successful
0929 20:25:52.829555 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2723,"thread_start_us":2603,"threads_started":1}
I20260929 20:25:52.830708 31759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:25:52.811027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:37213 (local address 127.17.53.65:57231)
0929 20:25:52.812019 (+ 992us) negotiation.cc:107] Waiting for socket to connect
0929 20:25:52.812067 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0929 20:25:52.813762 (+ 1695us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:25:52.814479 (+ 717us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:25:52.814487 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:25:52.815099 (+ 612us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:25:52.815661 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:25:52.815693 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:25:52.817211 (+ 1518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:25:52.817215 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:25:52.818432 (+ 1217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:25:52.818443 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:25:52.819531 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:25:52.819536 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:25:52.819735 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:25:52.820418 (+ 683us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:25:52.820441 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:25:52.822683 (+ 2242us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:25:52.826724 (+ 4041us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:25:52.826732 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:25:52.826746 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:25:52.827258 (+ 512us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:25:52.827650 (+ 392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:25:52.827653 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:25:52.827655 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:25:52.827866 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:25:52.828524 (+ 658us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:25:52.828530 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:25:52.828887 (+ 357us) client_negotiation.cc:769] Sending connection context
0929 20:25:52.829115 (+ 228us) client_negotiation.cc:240] Negotiation successful
0929 20:25:52.829529 (+ 414us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":694,"thread_start_us":384,"threads_started":1}
I20260929 20:25:52.832468 31756 heartbeater.cc:347] Connected to a master server at 127.17.53.126:37213
I20260929 20:25:52.832795 31756 heartbeater.cc:464] Registering TS with master...
I20260929 20:25:52.833638 31756 heartbeater.cc:511] Master 127.17.53.126:37213 requested a full tablet report, sending...
I20260929 20:25:52.836267 31567 ts_manager.cc:194] Registered new tserver with Master: ad1b6c08928b446ca0b8539b56223f57 (127.17.53.65:36129)
WARNING: no policy specified for kudu/127.17.53.66@KRBTEST.COM; defaulting to no policy
I20260929 20:25:52.838150 31567 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.53.65@KRBTEST.COM'} at 127.17.53.65:57231
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.53.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.53.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.66.keytab.
Entry for principal kudu/127.17.53.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.66.keytab.
I20260929 20:25:52.870393 17621 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
/tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713339161492-17621-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713339161492-17621-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713339161492-17621-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713339161492-17621-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.53.66:0
--local_ip_for_outbound_sockets=127.17.53.66
--webserver_interface=127.17.53.66
--webserver_port=0
--tserver_master_addrs=127.17.53.126:37213
--keytab_file=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.66.keytab
--principal=kudu/127.17.53.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.53.84:41567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713339161492-17621-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713339161492-17621-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713339161492-17621-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:25:53.196039 31762 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:25:53.196329 31762 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:25:53.196424 31762 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:25:53.203889 31762 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:25:53.203994 31762 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:25:53.204106 31762 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.53.66
I20260929 20:25:53.213914 31762 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.53.84:41567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713339161492-17621-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713339161492-17621-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.66.keytab
--principal=kudu/127.17.53.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.53.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713339161492-17621-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.17.53.66
--webserver_port=0
--tserver_master_addrs=127.17.53.126:37213
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.53.66
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713339161492-17621-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:25:53.215235 31762 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:25:53.216490 31762 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:25:53.229663 31767 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:25:53.229664 31770 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:25:53.229893 31768 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:25:53.230252 31762 server_base.cc:1034] running on GCE node
I20260929 20:25:53.230911 31762 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:25:53.231842 31762 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:25:53.233042 31762 hybrid_clock.cc:648] HybridClock initialized: now 1790713553232993 us; error 41 us; skew 500 ppm
I20260929 20:25:53.233366 31762 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:25:53 dist-test-slave-zbnc krb5kdc[31522](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713553, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.53.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:25:53.236605 31762 init.cc:377] Logged in from keytab as kudu/127.17.53.66@KRBTEST.COM (short username kudu)
I20260929 20:25:53.238018 31762 webserver.cc:460] Webserver started at http://127.17.53.66:45857/ using document root <none> and password file <none>
I20260929 20:25:53.238695 31762 fs_manager.cc:359] Metadata directory not provided
I20260929 20:25:53.238832 31762 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:25:53.239261 31762 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:25:53.242081 31762 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713339161492-17621-0/minicluster-data/ts-1/data/instance:
uuid: "eedca80ed76e4368a624213c9faed862"
format_stamp: "Formatted at 2026-09-29 20:25:53 on dist-test-slave-zbnc"
server_key: "7b18e8c3cf0d8876f9869e66f8cc41a4"
server_key_iv: "5a1d701cc0dad48f1c12dd09c3debb72"
server_key_version: "encryptionkey@0"
I20260929 20:25:53.242913 31762 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713339161492-17621-0/minicluster-data/ts-1/wal/instance:
uuid: "eedca80ed76e4368a624213c9faed862"
format_stamp: "Formatted at 2026-09-29 20:25:53 on dist-test-slave-zbnc"
server_key: "7b18e8c3cf0d8876f9869e66f8cc41a4"
server_key_iv: "5a1d701cc0dad48f1c12dd09c3debb72"
server_key_version: "encryptionkey@0"
I20260929 20:25:53.248258 31762 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260929 20:25:53.251596 31777 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:25:53.252884 31762 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 20:25:53.253060 31762 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713339161492-17621-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713339161492-17621-0/minicluster-data/ts-1/wal
uuid: "eedca80ed76e4368a624213c9faed862"
format_stamp: "Formatted at 2026-09-29 20:25:53 on dist-test-slave-zbnc"
server_key: "7b18e8c3cf0d8876f9869e66f8cc41a4"
server_key_iv: "5a1d701cc0dad48f1c12dd09c3debb72"
server_key_version: "encryptionkey@0"
I20260929 20:25:53.253259 31762 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713339161492-17621-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713339161492-17621-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713339161492-17621-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:25:53.272433 31762 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:25:53.276824 31762 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:25:53.277120 31762 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:25:53.278061 31762 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:25:53.279451 31762 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:25:53.279554 31762 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:25:53.279704 31762 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:25:53.279778 31762 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:25:53.313870 31762 rpc_server.cc:307] RPC server started. Bound to: 127.17.53.66:43527
I20260929 20:25:53.313957 31889 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.53.66:43527 every 8 connection(s)
I20260929 20:25:53.315284 31762 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713339161492-17621-0/minicluster-data/ts-1/data/info.pb
I20260929 20:25:53.323014 17621 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu as pid 31762
I20260929 20:25:53.323202 17621 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713339161492-17621-0/minicluster-data/ts-1/wal/instance
I20260929 20:25:53.323755 17621 external_mini_cluster.cc:1428] Setting key 5132c2e9e527a25cd3acb44cd2e66b8e
Sep 29 20:25:53 dist-test-slave-zbnc krb5kdc[31522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713553, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.53.66@KRBTEST.COM for kudu/127.17.53.126@KRBTEST.COM
I20260929 20:25:53.336944 31758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:25:53.317934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.53.66:56825 (local address 127.17.53.126:37213)
0929 20:25:53.318143 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0929 20:25:53.318148 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:25:53.319460 (+ 1312us) server_negotiation.cc:407] Connection header received
0929 20:25:53.321281 (+ 1821us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:25:53.321286 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:25:53.321383 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:25:53.321497 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:25:53.323333 (+ 1836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:25:53.324386 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:25:53.325915 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:25:53.326350 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:25:53.326478 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:25:53.329904 (+ 3426us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:25:53.329912 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:25:53.329921 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:25:53.329969 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:25:53.333484 (+ 3515us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:25:53.334562 (+ 1078us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:25:53.334566 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:25:53.334568 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:25:53.334698 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:25:53.335230 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:25:53.335234 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:25:53.335236 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:25:53.335502 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:25:53.335622 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0929 20:25:53.336503 (+ 881us) server_negotiation.cc:299] Negotiation successful
0929 20:25:53.336681 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260929 20:25:53.338578 31892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:25:53.318524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:37213 (local address 127.17.53.66:56825)
0929 20:25:53.319299 (+ 775us) negotiation.cc:107] Waiting for socket to connect
0929 20:25:53.319343 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 20:25:53.321001 (+ 1658us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:25:53.321755 (+ 754us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:25:53.321764 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:25:53.322286 (+ 522us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:25:53.323113 (+ 827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:25:53.323142 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:25:53.324538 (+ 1396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:25:53.324543 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:25:53.325766 (+ 1223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:25:53.325777 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:25:53.326581 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:25:53.326588 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:25:53.326832 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:25:53.327558 (+ 726us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:25:53.327585 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:25:53.329672 (+ 2087us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:25:53.333630 (+ 3958us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:25:53.333640 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:25:53.333661 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:25:53.334345 (+ 684us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:25:53.334836 (+ 491us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:25:53.334845 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:25:53.334851 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:25:53.335114 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:25:53.335625 (+ 511us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:25:53.335637 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:25:53.336116 (+ 479us) client_negotiation.cc:769] Sending connection context
0929 20:25:53.336421 (+ 305us) client_negotiation.cc:240] Negotiation successful
0929 20:25:53.337086 (+ 665us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":529,"thread_start_us":332,"threads_started":1}
I20260929 20:25:53.340087 31890 heartbeater.cc:347] Connected to a master server at 127.17.53.126:37213
I20260929 20:25:53.340400 31890 heartbeater.cc:464] Registering TS with master...
I20260929 20:25:53.341171 31890 heartbeater.cc:511] Master 127.17.53.126:37213 requested a full tablet report, sending...
I20260929 20:25:53.342633 31567 ts_manager.cc:194] Registered new tserver with Master: eedca80ed76e4368a624213c9faed862 (127.17.53.66:43527)
I20260929 20:25:53.343504 31567 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.53.66@KRBTEST.COM'} at 127.17.53.66:56825
WARNING: no policy specified for kudu/127.17.53.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.53.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.53.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.67.keytab.
Entry for principal kudu/127.17.53.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.67.keytab.
I20260929 20:25:53.378388 17621 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
/tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713339161492-17621-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713339161492-17621-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713339161492-17621-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713339161492-17621-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.53.67:0
--local_ip_for_outbound_sockets=127.17.53.67
--webserver_interface=127.17.53.67
--webserver_port=0
--tserver_master_addrs=127.17.53.126:37213
--keytab_file=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.67.keytab
--principal=kudu/127.17.53.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.53.84:41567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713339161492-17621-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713339161492-17621-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713339161492-17621-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:25:53.708016 31895 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:25:53.708315 31895 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:25:53.708410 31895 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:25:53.716022 31895 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:25:53.716146 31895 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:25:53.716257 31895 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.53.67
I20260929 20:25:53.726238 31895 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.53.84:41567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713339161492-17621-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713339161492-17621-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.67.keytab
--principal=kudu/127.17.53.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.53.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713339161492-17621-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.17.53.67
--webserver_port=0
--tserver_master_addrs=127.17.53.126:37213
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.53.67
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713339161492-17621-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:25:53.727713 31895 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:25:53.729038 31895 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:25:53.742895 31901 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:25:53.742935 31903 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:25:53.742959 31900 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:25:53.743615 31895 server_base.cc:1034] running on GCE node
I20260929 20:25:53.744264 31895 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:25:53.745471 31895 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:25:53.746721 31895 hybrid_clock.cc:648] HybridClock initialized: now 1790713553746606 us; error 111 us; skew 500 ppm
I20260929 20:25:53.747073 31895 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:25:53 dist-test-slave-zbnc krb5kdc[31522](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713553, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.53.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:25:53.750286 31895 init.cc:377] Logged in from keytab as kudu/127.17.53.67@KRBTEST.COM (short username kudu)
I20260929 20:25:53.751854 31895 webserver.cc:460] Webserver started at http://127.17.53.67:40081/ using document root <none> and password file <none>
I20260929 20:25:53.752557 31895 fs_manager.cc:359] Metadata directory not provided
I20260929 20:25:53.752729 31895 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:25:53.753053 31895 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:25:53.756173 31895 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713339161492-17621-0/minicluster-data/ts-2/data/instance:
uuid: "77871fa252864a14a60aefdf7b95e58d"
format_stamp: "Formatted at 2026-09-29 20:25:53 on dist-test-slave-zbnc"
server_key: "283af3aeaffef06b486bfbab35cd9520"
server_key_iv: "92e1a9b45c86a47eae578b5751b56ce6"
server_key_version: "encryptionkey@0"
I20260929 20:25:53.757041 31895 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713339161492-17621-0/minicluster-data/ts-2/wal/instance:
uuid: "77871fa252864a14a60aefdf7b95e58d"
format_stamp: "Formatted at 2026-09-29 20:25:53 on dist-test-slave-zbnc"
server_key: "283af3aeaffef06b486bfbab35cd9520"
server_key_iv: "92e1a9b45c86a47eae578b5751b56ce6"
server_key_version: "encryptionkey@0"
I20260929 20:25:53.762382 31895 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260929 20:25:53.766036 31910 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:25:53.767448 31895 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 20:25:53.767619 31895 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713339161492-17621-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713339161492-17621-0/minicluster-data/ts-2/wal
uuid: "77871fa252864a14a60aefdf7b95e58d"
format_stamp: "Formatted at 2026-09-29 20:25:53 on dist-test-slave-zbnc"
server_key: "283af3aeaffef06b486bfbab35cd9520"
server_key_iv: "92e1a9b45c86a47eae578b5751b56ce6"
server_key_version: "encryptionkey@0"
I20260929 20:25:53.767840 31895 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713339161492-17621-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713339161492-17621-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713339161492-17621-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:25:53.799809 31895 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:25:53.804251 31895 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:25:53.804554 31895 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:25:53.805688 31895 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:25:53.807288 31895 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:25:53.807431 31895 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:25:53.807592 31895 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:25:53.807670 31895 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:25:53.841434 31895 rpc_server.cc:307] RPC server started. Bound to: 127.17.53.67:39129
I20260929 20:25:53.841490 32022 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.53.67:39129 every 8 connection(s)
I20260929 20:25:53.842002 31756 heartbeater.cc:503] Master 127.17.53.126:37213 was elected leader, sending a full tablet report...
I20260929 20:25:53.842983 31895 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713339161492-17621-0/minicluster-data/ts-2/data/info.pb
I20260929 20:25:53.851687 17621 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu as pid 31895
I20260929 20:25:53.851910 17621 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713339161492-17621-0/minicluster-data/ts-2/wal/instance
I20260929 20:25:53.852355 17621 external_mini_cluster.cc:1428] Setting key 0210d98485d4da416241d1811fe7bf0a
Sep 29 20:25:53 dist-test-slave-zbnc krb5kdc[31522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713553, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.53.67@KRBTEST.COM for kudu/127.17.53.126@KRBTEST.COM
I20260929 20:25:53.863217 32025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:25:53.845674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.53.67:49983 (local address 127.17.53.126:37213)
0929 20:25:53.846152 (+ 478us) server_negotiation.cc:206] Beginning negotiation
0929 20:25:53.846158 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:25:53.847379 (+ 1221us) server_negotiation.cc:407] Connection header received
0929 20:25:53.849128 (+ 1749us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:25:53.849132 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:25:53.849198 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:25:53.849292 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:25:53.850731 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:25:53.851544 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:25:53.853392 (+ 1848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:25:53.853909 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:25:53.854071 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:25:53.857646 (+ 3575us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:25:53.857651 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:25:53.857654 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:25:53.857709 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:25:53.860204 (+ 2495us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:25:53.860899 (+ 695us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:25:53.860906 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:25:53.860910 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:25:53.861032 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:25:53.861462 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:25:53.861468 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:25:53.861472 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:25:53.861744 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:25:53.861902 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0929 20:25:53.862584 (+ 682us) server_negotiation.cc:299] Negotiation successful
0929 20:25:53.862811 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":357,"thread_start_us":209,"threads_started":1}
I20260929 20:25:53.864233 32026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:25:53.846394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:37213 (local address 127.17.53.67:49983)
0929 20:25:53.847217 (+ 823us) negotiation.cc:107] Waiting for socket to connect
0929 20:25:53.847267 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0929 20:25:53.848883 (+ 1616us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:25:53.849491 (+ 608us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:25:53.849499 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:25:53.850021 (+ 522us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:25:53.850583 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:25:53.850599 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:25:53.851706 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:25:53.851711 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:25:53.853252 (+ 1541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:25:53.853263 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:25:53.854138 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:25:53.854147 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:25:53.854396 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:25:53.855111 (+ 715us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:25:53.855143 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:25:53.857433 (+ 2290us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:25:53.860346 (+ 2913us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:25:53.860355 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:25:53.860372 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:25:53.860778 (+ 406us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:25:53.861151 (+ 373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:25:53.861156 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:25:53.861160 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:25:53.861351 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:25:53.861874 (+ 523us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:25:53.861881 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:25:53.862215 (+ 334us) client_negotiation.cc:769] Sending connection context
0929 20:25:53.862493 (+ 278us) client_negotiation.cc:240] Negotiation successful
0929 20:25:53.863034 (+ 541us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":584,"thread_start_us":370,"threads_started":1}
I20260929 20:25:53.865587 32023 heartbeater.cc:347] Connected to a master server at 127.17.53.126:37213
I20260929 20:25:53.865891 32023 heartbeater.cc:464] Registering TS with master...
I20260929 20:25:53.866752 32023 heartbeater.cc:511] Master 127.17.53.126:37213 requested a full tablet report, sending...
I20260929 20:25:53.868291 31567 ts_manager.cc:194] Registered new tserver with Master: 77871fa252864a14a60aefdf7b95e58d (127.17.53.67:39129)
I20260929 20:25:53.869254 31567 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.53.67@KRBTEST.COM'} at 127.17.53.67:49983
I20260929 20:25:53.880015 17621 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 20:25:53.897544 32025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:25:53.884533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53012 (local address 127.17.53.126:37213)
0929 20:25:53.884697 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0929 20:25:53.884703 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:25:53.885119 (+ 416us) server_negotiation.cc:407] Connection header received
0929 20:25:53.885287 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:25:53.885291 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:25:53.885371 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:25:53.885459 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:25:53.886837 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:25:53.887622 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:25:53.888992 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:25:53.889405 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:25:53.889545 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:25:53.892136 (+ 2591us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:25:53.892140 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:25:53.892144 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:25:53.892178 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:25:53.894712 (+ 2534us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:25:53.895468 (+ 756us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:25:53.895474 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:25:53.895478 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:25:53.895611 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:25:53.896005 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:25:53.896010 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:25:53.896013 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:25:53.896357 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:25:53.896613 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0929 20:25:53.896968 (+ 355us) server_negotiation.cc:299] Negotiation successful
0929 20:25:53.897213 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 20:25:53.901806 31567 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53012:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 20:25:53.909348 31567 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 20:25:53.932818 32036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:25:53.922076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.67:39129 (local address 127.0.0.1:54622)
0929 20:25:53.923299 (+ 1223us) negotiation.cc:107] Waiting for socket to connect
0929 20:25:53.923351 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0929 20:25:53.923562 (+ 211us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:25:53.924994 (+ 1432us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:25:53.925003 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:25:53.925033 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:25:53.925476 (+ 443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:25:53.925502 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:25:53.927859 (+ 2357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:25:53.927865 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:25:53.930262 (+ 2397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:25:53.930276 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:25:53.931706 (+ 1430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:25:53.931713 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:25:53.931901 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:25:53.931958 (+ 57us) client_negotiation.cc:769] Sending connection context
0929 20:25:53.932100 (+ 142us) client_negotiation.cc:240] Negotiation successful
0929 20:25:53.932290 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":849,"mutex_wait_us":4,"spinlock_wait_cycles":14208,"thread_start_us":832,"threads_started":1}
I20260929 20:25:53.933342 32040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:25:53.922433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.66:43527 (local address 127.0.0.1:58290)
0929 20:25:53.924540 (+ 2107us) negotiation.cc:107] Waiting for socket to connect
0929 20:25:53.924564 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0929 20:25:53.924706 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:25:53.926389 (+ 1683us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:25:53.926396 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:25:53.926421 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:25:53.926896 (+ 475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:25:53.926912 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:25:53.928542 (+ 1630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:25:53.928551 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:25:53.930572 (+ 2021us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:25:53.930587 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:25:53.932465 (+ 1878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:25:53.932471 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:25:53.932644 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:25:53.932675 (+ 31us) client_negotiation.cc:769] Sending connection context
0929 20:25:53.932769 (+ 94us) client_negotiation.cc:240] Negotiation successful
0929 20:25:53.932935 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1956,"mutex_wait_us":21,"thread_start_us":3439,"threads_started":1}
I20260929 20:25:53.933813 32038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:25:53.922401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54622 (local address 127.17.53.67:39129)
0929 20:25:53.924267 (+ 1866us) server_negotiation.cc:206] Beginning negotiation
0929 20:25:53.924275 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 20:25:53.924307 (+ 32us) server_negotiation.cc:407] Connection header received
0929 20:25:53.924410 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:25:53.924420 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:25:53.924730 (+ 310us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:25:53.925739 (+ 1009us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:25:53.926033 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:25:53.927725 (+ 1692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:25:53.930424 (+ 2699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:25:53.931513 (+ 1089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:25:53.931678 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:25:53.931934 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0929 20:25:53.932158 (+ 224us) server_negotiation.cc:299] Negotiation successful
0929 20:25:53.932329 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1687,"thread_start_us":1501,"threads_started":1}
I20260929 20:25:53.934512 32037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:25:53.923888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.65:36129 (local address 127.0.0.1:39602)
0929 20:25:53.926121 (+ 2233us) negotiation.cc:107] Waiting for socket to connect
0929 20:25:53.926142 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 20:25:53.926350 (+ 208us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:25:53.927112 (+ 762us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:25:53.927116 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:25:53.927137 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:25:53.927479 (+ 342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:25:53.927488 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:25:53.929342 (+ 1854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:25:53.929348 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:25:53.931448 (+ 2100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:25:53.931462 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:25:53.933663 (+ 2201us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:25:53.933672 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:25:53.933870 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:25:53.933905 (+ 35us) client_negotiation.cc:769] Sending connection context
0929 20:25:53.934070 (+ 165us) client_negotiation.cc:240] Negotiation successful
0929 20:25:53.934196 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2120,"thread_start_us":357,"threads_started":1}
I20260929 20:25:53.934610 32041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:25:53.924257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39602 (local address 127.17.53.65:36129)
0929 20:25:53.925366 (+ 1109us) server_negotiation.cc:206] Beginning negotiation
0929 20:25:53.925377 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 20:25:53.926240 (+ 863us) server_negotiation.cc:407] Connection header received
0929 20:25:53.926533 (+ 293us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:25:53.926543 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:25:53.926755 (+ 212us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:25:53.926976 (+ 221us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:25:53.927636 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:25:53.929156 (+ 1520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:25:53.931613 (+ 2457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:25:53.933109 (+ 1496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:25:53.933277 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:25:53.933502 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0929 20:25:53.934122 (+ 620us) server_negotiation.cc:299] Negotiation successful
0929 20:25:53.934288 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":899,"thread_start_us":718,"threads_started":1}
I20260929 20:25:53.935938 32039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:25:53.922727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58290 (local address 127.17.53.66:43527)
0929 20:25:53.925797 (+ 3070us) server_negotiation.cc:206] Beginning negotiation
0929 20:25:53.925808 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 20:25:53.925839 (+ 31us) server_negotiation.cc:407] Connection header received
0929 20:25:53.925943 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:25:53.925951 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:25:53.926186 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:25:53.927011 (+ 825us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:25:53.927229 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:25:53.928405 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:25:53.930736 (+ 2331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:25:53.932235 (+ 1499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:25:53.933787 (+ 1552us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:25:53.935574 (+ 1787us) server_negotiation.cc:1035] Waiting for connection context
0929 20:25:53.935633 (+ 59us) server_negotiation.cc:299] Negotiation successful
0929 20:25:53.935747 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2906,"thread_start_us":2810,"threads_started":1}
I20260929 20:25:53.937808 31958 tablet_service.cc:1467] Processing CreateTablet for tablet e55c73e3fe154a1cbfe9f3fd6dc4360e (DEFAULT_TABLE table=test-table [id=06c6283b9d834e29ac38e46509f08bcf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:25:53.938009 31691 tablet_service.cc:1467] Processing CreateTablet for tablet e55c73e3fe154a1cbfe9f3fd6dc4360e (DEFAULT_TABLE table=test-table [id=06c6283b9d834e29ac38e46509f08bcf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:25:53.939360 31958 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e55c73e3fe154a1cbfe9f3fd6dc4360e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:25:53.939360 31691 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e55c73e3fe154a1cbfe9f3fd6dc4360e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:25:53.939144 31825 tablet_service.cc:1467] Processing CreateTablet for tablet e55c73e3fe154a1cbfe9f3fd6dc4360e (DEFAULT_TABLE table=test-table [id=06c6283b9d834e29ac38e46509f08bcf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:25:53.940299 31825 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e55c73e3fe154a1cbfe9f3fd6dc4360e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:25:53.951807 32042 tablet_bootstrap.cc:492] T e55c73e3fe154a1cbfe9f3fd6dc4360e P eedca80ed76e4368a624213c9faed862: Bootstrap starting.
I20260929 20:25:53.953981 32044 tablet_bootstrap.cc:492] T e55c73e3fe154a1cbfe9f3fd6dc4360e P ad1b6c08928b446ca0b8539b56223f57: Bootstrap starting.
I20260929 20:25:53.954768 32043 tablet_bootstrap.cc:492] T e55c73e3fe154a1cbfe9f3fd6dc4360e P 77871fa252864a14a60aefdf7b95e58d: Bootstrap starting.
I20260929 20:25:53.955987 32042 tablet_bootstrap.cc:654] T e55c73e3fe154a1cbfe9f3fd6dc4360e P eedca80ed76e4368a624213c9faed862: Neither blocks nor log segments found. Creating new log.
I20260929 20:25:53.957443 32042 log.cc:824] T e55c73e3fe154a1cbfe9f3fd6dc4360e P eedca80ed76e4368a624213c9faed862: Log is configured to *not* fsync() on all Append() calls
I20260929 20:25:53.958369 32043 tablet_bootstrap.cc:654] T e55c73e3fe154a1cbfe9f3fd6dc4360e P 77871fa252864a14a60aefdf7b95e58d: Neither blocks nor log segments found. Creating new log.
I20260929 20:25:53.958442 32044 tablet_bootstrap.cc:654] T e55c73e3fe154a1cbfe9f3fd6dc4360e P ad1b6c08928b446ca0b8539b56223f57: Neither blocks nor log segments found. Creating new log.
I20260929 20:25:53.959627 32043 log.cc:824] T e55c73e3fe154a1cbfe9f3fd6dc4360e P 77871fa252864a14a60aefdf7b95e58d: Log is configured to *not* fsync() on all Append() calls
I20260929 20:25:53.959697 32044 log.cc:824] T e55c73e3fe154a1cbfe9f3fd6dc4360e P ad1b6c08928b446ca0b8539b56223f57: Log is configured to *not* fsync() on all Append() calls
I20260929 20:25:53.960825 32042 tablet_bootstrap.cc:492] T e55c73e3fe154a1cbfe9f3fd6dc4360e P eedca80ed76e4368a624213c9faed862: No bootstrap required, opened a new log
I20260929 20:25:53.961113 32042 ts_tablet_manager.cc:1397] T e55c73e3fe154a1cbfe9f3fd6dc4360e P eedca80ed76e4368a624213c9faed862: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20260929 20:25:53.962571 32043 tablet_bootstrap.cc:492] T e55c73e3fe154a1cbfe9f3fd6dc4360e P 77871fa252864a14a60aefdf7b95e58d: No bootstrap required, opened a new log
I20260929 20:25:53.962646 32044 tablet_bootstrap.cc:492] T e55c73e3fe154a1cbfe9f3fd6dc4360e P ad1b6c08928b446ca0b8539b56223f57: No bootstrap required, opened a new log
I20260929 20:25:53.962837 32043 ts_tablet_manager.cc:1397] T e55c73e3fe154a1cbfe9f3fd6dc4360e P 77871fa252864a14a60aefdf7b95e58d: Time spent bootstrapping tablet: real 0.008s user 0.008s sys 0.000s
I20260929 20:25:53.962886 32044 ts_tablet_manager.cc:1397] T e55c73e3fe154a1cbfe9f3fd6dc4360e P ad1b6c08928b446ca0b8539b56223f57: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.000s
I20260929 20:25:53.968786 32042 raft_consensus.cc:348] T e55c73e3fe154a1cbfe9f3fd6dc4360e P eedca80ed76e4368a624213c9faed862 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77871fa252864a14a60aefdf7b95e58d" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 39129 } } peers { permanent_uuid: "eedca80ed76e4368a624213c9faed862" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 43527 } } peers { permanent_uuid: "ad1b6c08928b446ca0b8539b56223f57" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 36129 } }
I20260929 20:25:53.969213 32042 raft_consensus.cc:374] T e55c73e3fe154a1cbfe9f3fd6dc4360e P eedca80ed76e4368a624213c9faed862 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:25:53.969343 32042 raft_consensus.cc:729] T e55c73e3fe154a1cbfe9f3fd6dc4360e P eedca80ed76e4368a624213c9faed862 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eedca80ed76e4368a624213c9faed862, State: Initialized, Role: FOLLOWER
I20260929 20:25:53.969995 32042 consensus_queue.cc:260] T e55c73e3fe154a1cbfe9f3fd6dc4360e P eedca80ed76e4368a624213c9faed862 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77871fa252864a14a60aefdf7b95e58d" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 39129 } } peers { permanent_uuid: "eedca80ed76e4368a624213c9faed862" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 43527 } } peers { permanent_uuid: "ad1b6c08928b446ca0b8539b56223f57" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 36129 } }
I20260929 20:25:53.971223 31890 heartbeater.cc:503] Master 127.17.53.126:37213 was elected leader, sending a full tablet report...
I20260929 20:25:53.971841 32042 ts_tablet_manager.cc:1428] T e55c73e3fe154a1cbfe9f3fd6dc4360e P eedca80ed76e4368a624213c9faed862: Time spent starting tablet: real 0.011s user 0.010s sys 0.000s
I20260929 20:25:53.973579 32043 raft_consensus.cc:348] T e55c73e3fe154a1cbfe9f3fd6dc4360e P 77871fa252864a14a60aefdf7b95e58d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77871fa252864a14a60aefdf7b95e58d" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 39129 } } peers { permanent_uuid: "eedca80ed76e4368a624213c9faed862" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 43527 } } peers { permanent_uuid: "ad1b6c08928b446ca0b8539b56223f57" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 36129 } }
I20260929 20:25:53.974009 32043 raft_consensus.cc:374] T e55c73e3fe154a1cbfe9f3fd6dc4360e P 77871fa252864a14a60aefdf7b95e58d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:25:53.974116 32043 raft_consensus.cc:729] T e55c73e3fe154a1cbfe9f3fd6dc4360e P 77871fa252864a14a60aefdf7b95e58d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77871fa252864a14a60aefdf7b95e58d, State: Initialized, Role: FOLLOWER
I20260929 20:25:53.974429 32044 raft_consensus.cc:348] T e55c73e3fe154a1cbfe9f3fd6dc4360e P ad1b6c08928b446ca0b8539b56223f57 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77871fa252864a14a60aefdf7b95e58d" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 39129 } } peers { permanent_uuid: "eedca80ed76e4368a624213c9faed862" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 43527 } } peers { permanent_uuid: "ad1b6c08928b446ca0b8539b56223f57" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 36129 } }
I20260929 20:25:53.974903 32044 raft_consensus.cc:374] T e55c73e3fe154a1cbfe9f3fd6dc4360e P ad1b6c08928b446ca0b8539b56223f57 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:25:53.975042 32044 raft_consensus.cc:729] T e55c73e3fe154a1cbfe9f3fd6dc4360e P ad1b6c08928b446ca0b8539b56223f57 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad1b6c08928b446ca0b8539b56223f57, State: Initialized, Role: FOLLOWER
I20260929 20:25:53.974720 32043 consensus_queue.cc:260] T e55c73e3fe154a1cbfe9f3fd6dc4360e P 77871fa252864a14a60aefdf7b95e58d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77871fa252864a14a60aefdf7b95e58d" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 39129 } } peers { permanent_uuid: "eedca80ed76e4368a624213c9faed862" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 43527 } } peers { permanent_uuid: "ad1b6c08928b446ca0b8539b56223f57" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 36129 } }
I20260929 20:25:53.975625 32044 consensus_queue.cc:260] T e55c73e3fe154a1cbfe9f3fd6dc4360e P ad1b6c08928b446ca0b8539b56223f57 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77871fa252864a14a60aefdf7b95e58d" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 39129 } } peers { permanent_uuid: "eedca80ed76e4368a624213c9faed862" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 43527 } } peers { permanent_uuid: "ad1b6c08928b446ca0b8539b56223f57" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 36129 } }
I20260929 20:25:53.976754 32023 heartbeater.cc:503] Master 127.17.53.126:37213 was elected leader, sending a full tablet report...
I20260929 20:25:53.977295 32043 ts_tablet_manager.cc:1428] T e55c73e3fe154a1cbfe9f3fd6dc4360e P 77871fa252864a14a60aefdf7b95e58d: Time spent starting tablet: real 0.014s user 0.009s sys 0.003s
I20260929 20:25:53.979444 32044 ts_tablet_manager.cc:1428] T e55c73e3fe154a1cbfe9f3fd6dc4360e P ad1b6c08928b446ca0b8539b56223f57: Time spent starting tablet: real 0.016s user 0.007s sys 0.008s
I20260929 20:25:54.009315 32048 raft_consensus.cc:482] T e55c73e3fe154a1cbfe9f3fd6dc4360e P eedca80ed76e4368a624213c9faed862 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 20:25:54.009590 32048 raft_consensus.cc:504] T e55c73e3fe154a1cbfe9f3fd6dc4360e P eedca80ed76e4368a624213c9faed862 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77871fa252864a14a60aefdf7b95e58d" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 39129 } } peers { permanent_uuid: "eedca80ed76e4368a624213c9faed862" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 43527 } } peers { permanent_uuid: "ad1b6c08928b446ca0b8539b56223f57" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 36129 } }
I20260929 20:25:54.011178 32048 leader_election.cc:288] T e55c73e3fe154a1cbfe9f3fd6dc4360e P eedca80ed76e4368a624213c9faed862 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 77871fa252864a14a60aefdf7b95e58d (127.17.53.67:39129), ad1b6c08928b446ca0b8539b56223f57 (127.17.53.65:36129)
I20260929 20:25:54.018416 32038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:25:54.011665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.53.66:35037 (local address 127.17.53.67:39129)
0929 20:25:54.011921 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0929 20:25:54.011930 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 20:25:54.012411 (+ 481us) server_negotiation.cc:407] Connection header received
0929 20:25:54.012637 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:25:54.012644 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:25:54.012754 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:25:54.012874 (+ 120us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:25:54.013594 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:25:54.014526 (+ 932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:25:54.016529 (+ 2003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:25:54.017414 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:25:54.017538 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:25:54.017591 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0929 20:25:54.018025 (+ 434us) server_negotiation.cc:299] Negotiation successful
0929 20:25:54.018155 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260929 20:25:54.018537 32051 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:25:54.011608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.67:39129 (local address 127.17.53.66:35037)
0929 20:25:54.012303 (+ 695us) negotiation.cc:107] Waiting for socket to connect
0929 20:25:54.012319 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 20:25:54.012491 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:25:54.012988 (+ 497us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:25:54.012992 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:25:54.013017 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:25:54.013448 (+ 431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:25:54.013463 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:25:54.014662 (+ 1199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:25:54.014668 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:25:54.016381 (+ 1713us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:25:54.016393 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:25:54.017639 (+ 1246us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:25:54.017646 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:25:54.017846 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:25:54.017885 (+ 39us) client_negotiation.cc:769] Sending connection context
0929 20:25:54.017980 (+ 95us) client_negotiation.cc:240] Negotiation successful
0929 20:25:54.018094 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":561,"mutex_wait_us":23,"spinlock_wait_cycles":768,"thread_start_us":404,"threads_started":1}
I20260929 20:25:54.018954 32052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:25:54.012392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.65:36129 (local address 127.17.53.66:35285)
0929 20:25:54.013070 (+ 678us) negotiation.cc:107] Waiting for socket to connect
0929 20:25:54.013084 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 20:25:54.013203 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:25:54.013581 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:25:54.013585 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:25:54.013608 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:25:54.013969 (+ 361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:25:54.013979 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:25:54.015166 (+ 1187us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:25:54.015169 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:25:54.016664 (+ 1495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:25:54.016677 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:25:54.017927 (+ 1250us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:25:54.017931 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:25:54.018138 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:25:54.018167 (+ 29us) client_negotiation.cc:769] Sending connection context
0929 20:25:54.018296 (+ 129us) client_negotiation.cc:240] Negotiation successful
0929 20:25:54.018416 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":547,"mutex_wait_us":21,"spinlock_wait_cycles":1792,"thread_start_us":350,"threads_started":1}
I20260929 20:25:54.018949 32041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:25:54.012663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.53.66:35285 (local address 127.17.53.65:36129)
0929 20:25:54.012901 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0929 20:25:54.012907 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:25:54.013179 (+ 272us) server_negotiation.cc:407] Connection header received
0929 20:25:54.013343 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:25:54.013348 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:25:54.013432 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:25:54.013539 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:25:54.014111 (+ 572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:25:54.015032 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:25:54.016817 (+ 1785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:25:54.017729 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:25:54.017886 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:25:54.017968 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0929 20:25:54.018323 (+ 355us) server_negotiation.cc:299] Negotiation successful
0929 20:25:54.018460 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260929 20:25:54.019739 31978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e55c73e3fe154a1cbfe9f3fd6dc4360e" candidate_uuid: "eedca80ed76e4368a624213c9faed862" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77871fa252864a14a60aefdf7b95e58d" is_pre_election: true
I20260929 20:25:54.019954 31711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e55c73e3fe154a1cbfe9f3fd6dc4360e" candidate_uuid: "eedca80ed76e4368a624213c9faed862" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad1b6c08928b446ca0b8539b56223f57" is_pre_election: true
I20260929 20:25:54.020382 31978 raft_consensus.cc:2454] T e55c73e3fe154a1cbfe9f3fd6dc4360e P 77871fa252864a14a60aefdf7b95e58d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eedca80ed76e4368a624213c9faed862 in term 0.
I20260929 20:25:54.020473 31711 raft_consensus.cc:2454] T e55c73e3fe154a1cbfe9f3fd6dc4360e P ad1b6c08928b446ca0b8539b56223f57 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eedca80ed76e4368a624213c9faed862 in term 0.
I20260929 20:25:54.021075 31781 leader_election.cc:302] T e55c73e3fe154a1cbfe9f3fd6dc4360e P eedca80ed76e4368a624213c9faed862 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ad1b6c08928b446ca0b8539b56223f57, eedca80ed76e4368a624213c9faed862; no voters:
I20260929 20:25:54.021420 32048 raft_consensus.cc:2781] T e55c73e3fe154a1cbfe9f3fd6dc4360e P eedca80ed76e4368a624213c9faed862 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 20:25:54.021574 32048 raft_consensus.cc:482] T e55c73e3fe154a1cbfe9f3fd6dc4360e P eedca80ed76e4368a624213c9faed862 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 20:25:54.021683 32048 raft_consensus.cc:3037] T e55c73e3fe154a1cbfe9f3fd6dc4360e P eedca80ed76e4368a624213c9faed862 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:25:54.024133 32048 raft_consensus.cc:504] T e55c73e3fe154a1cbfe9f3fd6dc4360e P eedca80ed76e4368a624213c9faed862 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77871fa252864a14a60aefdf7b95e58d" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 39129 } } peers { permanent_uuid: "eedca80ed76e4368a624213c9faed862" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 43527 } } peers { permanent_uuid: "ad1b6c08928b446ca0b8539b56223f57" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 36129 } }
I20260929 20:25:54.024794 32048 leader_election.cc:288] T e55c73e3fe154a1cbfe9f3fd6dc4360e P eedca80ed76e4368a624213c9faed862 [CANDIDATE]: Term 1 election: Requested vote from peers 77871fa252864a14a60aefdf7b95e58d (127.17.53.67:39129), ad1b6c08928b446ca0b8539b56223f57 (127.17.53.65:36129)
I20260929 20:25:54.025050 31978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e55c73e3fe154a1cbfe9f3fd6dc4360e" candidate_uuid: "eedca80ed76e4368a624213c9faed862" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "77871fa252864a14a60aefdf7b95e58d"
I20260929 20:25:54.025240 31978 raft_consensus.cc:3037] T e55c73e3fe154a1cbfe9f3fd6dc4360e P 77871fa252864a14a60aefdf7b95e58d [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:25:54.025274 31711 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e55c73e3fe154a1cbfe9f3fd6dc4360e" candidate_uuid: "eedca80ed76e4368a624213c9faed862" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad1b6c08928b446ca0b8539b56223f57"
I20260929 20:25:54.025491 31711 raft_consensus.cc:3037] T e55c73e3fe154a1cbfe9f3fd6dc4360e P ad1b6c08928b446ca0b8539b56223f57 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:25:54.027634 31978 raft_consensus.cc:2454] T e55c73e3fe154a1cbfe9f3fd6dc4360e P 77871fa252864a14a60aefdf7b95e58d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eedca80ed76e4368a624213c9faed862 in term 1.
I20260929 20:25:54.027738 31711 raft_consensus.cc:2454] T e55c73e3fe154a1cbfe9f3fd6dc4360e P ad1b6c08928b446ca0b8539b56223f57 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eedca80ed76e4368a624213c9faed862 in term 1.
I20260929 20:25:54.028080 31781 leader_election.cc:302] T e55c73e3fe154a1cbfe9f3fd6dc4360e P eedca80ed76e4368a624213c9faed862 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77871fa252864a14a60aefdf7b95e58d, eedca80ed76e4368a624213c9faed862; no voters:
I20260929 20:25:54.028388 32048 raft_consensus.cc:2781] T e55c73e3fe154a1cbfe9f3fd6dc4360e P eedca80ed76e4368a624213c9faed862 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:25:54.028839 32048 raft_consensus.cc:686] T e55c73e3fe154a1cbfe9f3fd6dc4360e P eedca80ed76e4368a624213c9faed862 [term 1 LEADER]: Becoming Leader. State: Replica: eedca80ed76e4368a624213c9faed862, State: Running, Role: LEADER
I20260929 20:25:54.029440 32048 consensus_queue.cc:237] T e55c73e3fe154a1cbfe9f3fd6dc4360e P eedca80ed76e4368a624213c9faed862 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77871fa252864a14a60aefdf7b95e58d" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 39129 } } peers { permanent_uuid: "eedca80ed76e4368a624213c9faed862" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 43527 } } peers { permanent_uuid: "ad1b6c08928b446ca0b8539b56223f57" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 36129 } }
I20260929 20:25:54.034046 31567 catalog_manager.cc:5495] T e55c73e3fe154a1cbfe9f3fd6dc4360e P eedca80ed76e4368a624213c9faed862 reported cstate change: term changed from 0 to 1, leader changed from <none> to eedca80ed76e4368a624213c9faed862 (127.17.53.66). New cstate: current_term: 1 leader_uuid: "eedca80ed76e4368a624213c9faed862" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "77871fa252864a14a60aefdf7b95e58d" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 39129 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eedca80ed76e4368a624213c9faed862" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 43527 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ad1b6c08928b446ca0b8539b56223f57" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 36129 } health_report { overall_health: UNKNOWN } } }
W20260929 20:25:54.060006 31757 tablet.cc:2369] T e55c73e3fe154a1cbfe9f3fd6dc4360e P ad1b6c08928b446ca0b8539b56223f57: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 20:25:54.061117 32039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:25:54.055365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58304 (local address 127.17.53.66:43527)
0929 20:25:54.055545 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0929 20:25:54.055550 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:25:54.055624 (+ 74us) server_negotiation.cc:407] Connection header received
0929 20:25:54.055817 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:25:54.055821 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:25:54.055906 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:25:54.056031 (+ 125us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 20:25:54.056522 (+ 491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:25:54.057253 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:25:54.058797 (+ 1544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:25:54.059238 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:25:54.059330 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:25:54.059844 (+ 514us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 20:25:54.060577 (+ 733us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 20:25:54.060733 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0929 20:25:54.060840 (+ 107us) server_negotiation.cc:299] Negotiation successful
0929 20:25:54.060915 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
W20260929 20:25:54.067202 31891 tablet.cc:2369] T e55c73e3fe154a1cbfe9f3fd6dc4360e P eedca80ed76e4368a624213c9faed862: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 20:25:54.069953 31711 raft_consensus.cc:1261] T e55c73e3fe154a1cbfe9f3fd6dc4360e P ad1b6c08928b446ca0b8539b56223f57 [term 1 FOLLOWER]: Refusing update from remote peer eedca80ed76e4368a624213c9faed862: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:25:54.070009 31978 raft_consensus.cc:1261] T e55c73e3fe154a1cbfe9f3fd6dc4360e P 77871fa252864a14a60aefdf7b95e58d [term 1 FOLLOWER]: Refusing update from remote peer eedca80ed76e4368a624213c9faed862: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:25:54.071030 32053 consensus_queue.cc:1035] T e55c73e3fe154a1cbfe9f3fd6dc4360e P eedca80ed76e4368a624213c9faed862 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad1b6c08928b446ca0b8539b56223f57" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 36129 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:25:54.071437 32048 consensus_queue.cc:1035] T e55c73e3fe154a1cbfe9f3fd6dc4360e P eedca80ed76e4368a624213c9faed862 [LEADER]: Connected to new peer: Peer: permanent_uuid: "77871fa252864a14a60aefdf7b95e58d" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 39129 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:25:54.082757 32056 mvcc.cc:205] Tried to move back new op lower bound from 7334762717458796544 to 7334762717308956672. Current Snapshot: MvccSnapshot[applied={T|T < 7334762717458796544}]
I20260929 20:25:54.100567 31567 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53012:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 20:25:54.100888 31567 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53012:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 20:25:54.104976 31567 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7caaae55b00c4da99832e09118f54e5e: Sending DeleteTablet for 3 replicas of tablet e55c73e3fe154a1cbfe9f3fd6dc4360e
I20260929 20:25:54.106238 31958 tablet_service.cc:1514] Processing DeleteTablet for tablet e55c73e3fe154a1cbfe9f3fd6dc4360e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:25:54 UTC) from {username='kudu', principal='kudu/127.17.53.126@KRBTEST.COM'} at 127.0.0.1:54622
I20260929 20:25:54.106259 31825 tablet_service.cc:1514] Processing DeleteTablet for tablet e55c73e3fe154a1cbfe9f3fd6dc4360e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:25:54 UTC) from {username='kudu', principal='kudu/127.17.53.126@KRBTEST.COM'} at 127.0.0.1:58290
I20260929 20:25:54.106683 31691 tablet_service.cc:1514] Processing DeleteTablet for tablet e55c73e3fe154a1cbfe9f3fd6dc4360e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:25:54 UTC) from {username='kudu', principal='kudu/127.17.53.126@KRBTEST.COM'} at 127.0.0.1:39602
I20260929 20:25:54.107501 17621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu pid 31628
I20260929 20:25:54.116125 32069 tablet_replica.cc:330] T e55c73e3fe154a1cbfe9f3fd6dc4360e P ad1b6c08928b446ca0b8539b56223f57: stopping tablet replica
I20260929 20:25:54.116880 32069 raft_consensus.cc:2229] T e55c73e3fe154a1cbfe9f3fd6dc4360e P ad1b6c08928b446ca0b8539b56223f57 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 20:25:54.117673 32069 raft_consensus.cc:2258] T e55c73e3fe154a1cbfe9f3fd6dc4360e P ad1b6c08928b446ca0b8539b56223f57 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:25:54.119526 32067 tablet_replica.cc:330] T e55c73e3fe154a1cbfe9f3fd6dc4360e P eedca80ed76e4368a624213c9faed862: stopping tablet replica
I20260929 20:25:54.121421 32069 ts_tablet_manager.cc:1905] T e55c73e3fe154a1cbfe9f3fd6dc4360e P ad1b6c08928b446ca0b8539b56223f57: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:25:54.122068 32067 raft_consensus.cc:2229] T e55c73e3fe154a1cbfe9f3fd6dc4360e P eedca80ed76e4368a624213c9faed862 [term 1 LEADER]: Raft consensus shutting down.
I20260929 20:25:54.127871 32067 raft_consensus.cc:2258] T e55c73e3fe154a1cbfe9f3fd6dc4360e P eedca80ed76e4368a624213c9faed862 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:25:54.129177 32068 tablet_replica.cc:330] T e55c73e3fe154a1cbfe9f3fd6dc4360e P 77871fa252864a14a60aefdf7b95e58d: stopping tablet replica
I20260929 20:25:54.129568 32068 raft_consensus.cc:2229] T e55c73e3fe154a1cbfe9f3fd6dc4360e P 77871fa252864a14a60aefdf7b95e58d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 20:25:54.130015 32068 raft_consensus.cc:2258] T e55c73e3fe154a1cbfe9f3fd6dc4360e P 77871fa252864a14a60aefdf7b95e58d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:25:54.132066 32067 ts_tablet_manager.cc:1905] T e55c73e3fe154a1cbfe9f3fd6dc4360e P eedca80ed76e4368a624213c9faed862: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:25:54.134738 32069 ts_tablet_manager.cc:1918] T e55c73e3fe154a1cbfe9f3fd6dc4360e P ad1b6c08928b446ca0b8539b56223f57: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 20:25:54.134994 32069 log.cc:1196] T e55c73e3fe154a1cbfe9f3fd6dc4360e P ad1b6c08928b446ca0b8539b56223f57: Deleting WAL directory at /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713339161492-17621-0/minicluster-data/ts-0/wal/wals/e55c73e3fe154a1cbfe9f3fd6dc4360e
Sep 29 20:25:54 dist-test-slave-zbnc krb5kdc[31522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713551, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.53.65@KRBTEST.COM
I20260929 20:25:54.135710 32069 ts_tablet_manager.cc:1939] T e55c73e3fe154a1cbfe9f3fd6dc4360e P ad1b6c08928b446ca0b8539b56223f57: Deleting consensus metadata
I20260929 20:25:54.137881 31554 catalog_manager.cc:4841] TS ad1b6c08928b446ca0b8539b56223f57 (127.17.53.65:36129): tablet e55c73e3fe154a1cbfe9f3fd6dc4360e (table test-table [id=06c6283b9d834e29ac38e46509f08bcf]) successfully deleted
I20260929 20:25:54.139408 32068 ts_tablet_manager.cc:1905] T e55c73e3fe154a1cbfe9f3fd6dc4360e P 77871fa252864a14a60aefdf7b95e58d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:25:54.144109 32068 ts_tablet_manager.cc:1918] T e55c73e3fe154a1cbfe9f3fd6dc4360e P 77871fa252864a14a60aefdf7b95e58d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 20:25:54.144299 32068 log.cc:1196] T e55c73e3fe154a1cbfe9f3fd6dc4360e P 77871fa252864a14a60aefdf7b95e58d: Deleting WAL directory at /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713339161492-17621-0/minicluster-data/ts-2/wal/wals/e55c73e3fe154a1cbfe9f3fd6dc4360e
I20260929 20:25:54.144824 32068 ts_tablet_manager.cc:1939] T e55c73e3fe154a1cbfe9f3fd6dc4360e P 77871fa252864a14a60aefdf7b95e58d: Deleting consensus metadata
I20260929 20:25:54.145615 32041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:25:54.108664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39616 (local address 127.17.53.65:36129)
0929 20:25:54.109999 (+ 1335us) server_negotiation.cc:206] Beginning negotiation
0929 20:25:54.110007 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 20:25:54.123826 (+ 13819us) server_negotiation.cc:407] Connection header received
0929 20:25:54.124328 (+ 502us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:25:54.124334 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:25:54.124421 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:25:54.124510 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:25:54.126557 (+ 2047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:25:54.127781 (+ 1224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:25:54.129842 (+ 2061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:25:54.130456 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:25:54.130550 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:25:54.136327 (+ 5777us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:25:54.136343 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:25:54.136361 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:25:54.136410 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:25:54.141272 (+ 4862us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:25:54.142450 (+ 1178us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:25:54.142461 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:25:54.142466 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:25:54.142665 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:25:54.143226 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:25:54.143232 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:25:54.143236 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:25:54.143732 (+ 496us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:25:54.144032 (+ 300us) server_negotiation.cc:1035] Waiting for connection context
0929 20:25:54.144765 (+ 733us) server_negotiation.cc:299] Negotiation successful
0929 20:25:54.145099 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1204}
I20260929 20:25:54.146733 31554 catalog_manager.cc:4841] TS 77871fa252864a14a60aefdf7b95e58d (127.17.53.67:39129): tablet e55c73e3fe154a1cbfe9f3fd6dc4360e (table test-table [id=06c6283b9d834e29ac38e46509f08bcf]) successfully deleted
I20260929 20:25:54.147006 32067 ts_tablet_manager.cc:1918] T e55c73e3fe154a1cbfe9f3fd6dc4360e P eedca80ed76e4368a624213c9faed862: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 20:25:54.147199 32067 log.cc:1196] T e55c73e3fe154a1cbfe9f3fd6dc4360e P eedca80ed76e4368a624213c9faed862: Deleting WAL directory at /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713339161492-17621-0/minicluster-data/ts-1/wal/wals/e55c73e3fe154a1cbfe9f3fd6dc4360e
I20260929 20:25:54.147280 31751 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:25:54.147656 32067 ts_tablet_manager.cc:1939] T e55c73e3fe154a1cbfe9f3fd6dc4360e P eedca80ed76e4368a624213c9faed862: Deleting consensus metadata
I20260929 20:25:54.149344 31551 catalog_manager.cc:4841] TS eedca80ed76e4368a624213c9faed862 (127.17.53.66:43527): tablet e55c73e3fe154a1cbfe9f3fd6dc4360e (table test-table [id=06c6283b9d834e29ac38e46509f08bcf]) successfully deleted
I20260929 20:25:54.242996 17621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu with pid 31628
I20260929 20:25:54.261250 17621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu pid 31762
Sep 29 20:25:54 dist-test-slave-zbnc krb5kdc[31522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713551, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.53.66@KRBTEST.COM
I20260929 20:25:54.281563 32039 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:25:54.262463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58308 (local address 127.17.53.66:43527)
0929 20:25:54.262723 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0929 20:25:54.262731 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 20:25:54.262764 (+ 33us) server_negotiation.cc:407] Connection header received
0929 20:25:54.262881 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:25:54.262889 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:25:54.262986 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:25:54.263131 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:25:54.264520 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:25:54.266014 (+ 1494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:25:54.267730 (+ 1716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:25:54.268549 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:25:54.268730 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:25:54.273455 (+ 4725us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:25:54.273475 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:25:54.273499 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:25:54.273562 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:25:54.278177 (+ 4615us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:25:54.278984 (+ 807us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:25:54.278996 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:25:54.279001 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:25:54.279199 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:25:54.279612 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:25:54.279618 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:25:54.279621 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:25:54.280142 (+ 521us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:25:54.280488 (+ 346us) server_negotiation.cc:1035] Waiting for connection context
0929 20:25:54.280739 (+ 251us) server_negotiation.cc:299] Negotiation successful
0929 20:25:54.280993 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260929 20:25:54.282552 31885 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:25:54.398370 17621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu with pid 31762
I20260929 20:25:54.414386 17621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu pid 31895
Sep 29 20:25:54 dist-test-slave-zbnc krb5kdc[31522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713551, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.53.67@KRBTEST.COM
I20260929 20:25:54.429219 32038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:25:54.415202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54628 (local address 127.17.53.67:39129)
0929 20:25:54.415369 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0929 20:25:54.415374 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:25:54.415392 (+ 18us) server_negotiation.cc:407] Connection header received
0929 20:25:54.415558 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:25:54.415562 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:25:54.415633 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:25:54.415718 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:25:54.416995 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:25:54.417716 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:25:54.419112 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:25:54.419498 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:25:54.419589 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:25:54.423929 (+ 4340us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:25:54.423943 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:25:54.423959 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:25:54.423992 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:25:54.426792 (+ 2800us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:25:54.427559 (+ 767us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:25:54.427567 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:25:54.427569 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:25:54.427700 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:25:54.428014 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:25:54.428017 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:25:54.428019 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:25:54.428357 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:25:54.428571 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0929 20:25:54.428803 (+ 232us) server_negotiation.cc:299] Negotiation successful
0929 20:25:54.428974 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 20:25:54.429903 32018 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:25:54.519577 17621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu with pid 31895
I20260929 20:25:54.536322 17621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu pid 31535
I20260929 20:25:54.537586 31597 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:25:54.649711 17621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu with pid 31535
I20260929 20:25:54.671118 17621 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
/tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713339161492-17621-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713339161492-17621-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713339161492-17621-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713339161492-17621-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.53.126:37213
--webserver_interface=127.17.53.126
--webserver_port=40845
--keytab_file=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.126.keytab
--principal=kudu/127.17.53.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.53.84:41567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.53.126:37213
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713339161492-17621-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713339161492-17621-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713339161492-17621-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:25:54.999577 32075 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:25:54.999948 32075 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:25:55.000118 32075 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:25:55.007531 32075 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:25:55.007694 32075 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:25:55.007807 32075 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:25:55.007913 32075 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:25:55.008008 32075 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:25:55.018242 32075 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.53.84:41567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713339161492-17621-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713339161492-17621-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.17.53.126:37213
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.126.keytab
--principal=kudu/127.17.53.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.53.126:37213
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713339161492-17621-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.53.126
--webserver_port=40845
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713339161492-17621-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:25:55.019666 32075 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:25:55.021027 32075 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:25:55.045903 32081 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:25:55.045990 32084 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:25:55.046011 32082 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:25:55.046880 32075 server_base.cc:1034] running on GCE node
I20260929 20:25:55.047803 32075 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:25:55.049506 32075 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:25:55.050774 32075 hybrid_clock.cc:648] HybridClock initialized: now 1790713555050684 us; error 58 us; skew 500 ppm
I20260929 20:25:55.051436 32075 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:25:55 dist-test-slave-zbnc krb5kdc[31522](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713555, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.53.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:25:55.059067 32075 init.cc:377] Logged in from keytab as kudu/127.17.53.126@KRBTEST.COM (short username kudu)
I20260929 20:25:55.064913 32075 webserver.cc:460] Webserver started at http://127.17.53.126:40845/ using document root <none> and password file <none>
I20260929 20:25:55.065949 32075 fs_manager.cc:359] Metadata directory not provided
I20260929 20:25:55.066141 32075 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:25:55.074713 32075 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.001s sys 0.005s
I20260929 20:25:55.079465 32091 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:25:55.081321 32075 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260929 20:25:55.081537 32075 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713339161492-17621-0/minicluster-data/master-0/data,/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713339161492-17621-0/minicluster-data/master-0/wal
uuid: "7caaae55b00c4da99832e09118f54e5e"
format_stamp: "Formatted at 2026-09-29 20:25:52 on dist-test-slave-zbnc"
server_key: "556bb7331b6e6cd015f30def5b023699"
server_key_iv: "e08504e502dda6de1daf7acf50e383a6"
server_key_version: "encryptionkey@0"
I20260929 20:25:55.083153 32075 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713339161492-17621-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713339161492-17621-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713339161492-17621-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:25:55.101015 32075 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:25:55.105867 32075 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:25:55.106176 32075 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:25:55.125656 32075 rpc_server.cc:307] RPC server started. Bound to: 127.17.53.126:37213
I20260929 20:25:55.125677 32142 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.53.126:37213 every 8 connection(s)
I20260929 20:25:55.127362 32075 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713339161492-17621-0/minicluster-data/master-0/data/info.pb
I20260929 20:25:55.134582 32143 sys_catalog.cc:263] Verifying existing consensus state
I20260929 20:25:55.134586 17621 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu as pid 32075
I20260929 20:25:55.135500 17621 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
/tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713339161492-17621-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713339161492-17621-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713339161492-17621-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713339161492-17621-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.53.65:36129
--local_ip_for_outbound_sockets=127.17.53.65
--tserver_master_addrs=127.17.53.126:37213
--webserver_port=46229
--webserver_interface=127.17.53.65
--keytab_file=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.65.keytab
--principal=kudu/127.17.53.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.53.84:41567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713339161492-17621-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713339161492-17621-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713339161492-17621-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 20:25:55.138046 32143 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7caaae55b00c4da99832e09118f54e5e: Bootstrap starting.
I20260929 20:25:55.157598 32143 log.cc:824] T 00000000000000000000000000000000 P 7caaae55b00c4da99832e09118f54e5e: Log is configured to *not* fsync() on all Append() calls
I20260929 20:25:55.169724 32143 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7caaae55b00c4da99832e09118f54e5e: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 20:25:55.170272 32143 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7caaae55b00c4da99832e09118f54e5e: Bootstrap complete.
I20260929 20:25:55.179622 32143 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7caaae55b00c4da99832e09118f54e5e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7caaae55b00c4da99832e09118f54e5e" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 37213 } }
I20260929 20:25:55.180485 32143 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7caaae55b00c4da99832e09118f54e5e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7caaae55b00c4da99832e09118f54e5e, State: Initialized, Role: FOLLOWER
I20260929 20:25:55.181128 32143 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7caaae55b00c4da99832e09118f54e5e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7caaae55b00c4da99832e09118f54e5e" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 37213 } }
I20260929 20:25:55.181459 32143 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7caaae55b00c4da99832e09118f54e5e [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 20:25:55.181602 32143 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7caaae55b00c4da99832e09118f54e5e [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 20:25:55.181804 32143 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7caaae55b00c4da99832e09118f54e5e [term 1 FOLLOWER]: Advancing to term 2
I20260929 20:25:55.185812 32143 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7caaae55b00c4da99832e09118f54e5e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7caaae55b00c4da99832e09118f54e5e" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 37213 } }
I20260929 20:25:55.186343 32143 leader_election.cc:302] T 00000000000000000000000000000000 P 7caaae55b00c4da99832e09118f54e5e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7caaae55b00c4da99832e09118f54e5e; no voters:
I20260929 20:25:55.187033 32143 leader_election.cc:288] T 00000000000000000000000000000000 P 7caaae55b00c4da99832e09118f54e5e [CANDIDATE]: Term 2 election: Requested vote from peers
I20260929 20:25:55.187175 32147 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7caaae55b00c4da99832e09118f54e5e [term 2 FOLLOWER]: Leader election won for term 2
I20260929 20:25:55.188691 32147 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7caaae55b00c4da99832e09118f54e5e [term 2 LEADER]: Becoming Leader. State: Replica: 7caaae55b00c4da99832e09118f54e5e, State: Running, Role: LEADER
I20260929 20:25:55.189358 32147 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7caaae55b00c4da99832e09118f54e5e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7caaae55b00c4da99832e09118f54e5e" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 37213 } }
I20260929 20:25:55.189761 32143 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7caaae55b00c4da99832e09118f54e5e [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:25:55.192008 32149 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7caaae55b00c4da99832e09118f54e5e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7caaae55b00c4da99832e09118f54e5e. Latest consensus state: current_term: 2 leader_uuid: "7caaae55b00c4da99832e09118f54e5e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7caaae55b00c4da99832e09118f54e5e" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 37213 } } }
I20260929 20:25:55.192382 32149 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7caaae55b00c4da99832e09118f54e5e [sys.catalog]: This master's current role is: LEADER
I20260929 20:25:55.192250 32148 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7caaae55b00c4da99832e09118f54e5e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "7caaae55b00c4da99832e09118f54e5e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7caaae55b00c4da99832e09118f54e5e" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 37213 } } }
I20260929 20:25:55.192796 32148 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7caaae55b00c4da99832e09118f54e5e [sys.catalog]: This master's current role is: LEADER
I20260929 20:25:55.201223 32153 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:25:55.214658 32153 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:25:55.216341 32153 catalog_manager.cc:1259] Loaded cluster ID: 4893bb9645e249e4b64ef32dfa3ad00f
I20260929 20:25:55.216465 32153 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:25:55.220242 32153 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260929 20:25:55.237262 32153 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260929 20:25:55.246433 32153 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:25:55.248557 32153 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 7caaae55b00c4da99832e09118f54e5e: Loaded TSK: 0
I20260929 20:25:55.250063 32153 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260929 20:25:55.489027 32145 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:25:55.489303 32145 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:25:55.489464 32145 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:25:55.497051 32145 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:25:55.497246 32145 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:25:55.497427 32145 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.53.65
I20260929 20:25:55.507534 32145 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.53.84:41567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713339161492-17621-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713339161492-17621-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.65.keytab
--principal=kudu/127.17.53.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.53.65:36129
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713339161492-17621-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.17.53.65
--webserver_port=46229
--tserver_master_addrs=127.17.53.126:37213
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.53.65
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713339161492-17621-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:25:55.508819 32145 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:25:55.510121 32145 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:25:55.523994 32174 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:25:55.523994 32172 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:25:55.524820 32171 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:25:55.525678 32145 server_base.cc:1034] running on GCE node
I20260929 20:25:55.526367 32145 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:25:55.527524 32145 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:25:55.528750 32145 hybrid_clock.cc:648] HybridClock initialized: now 1790713555528683 us; error 61 us; skew 500 ppm
I20260929 20:25:55.529071 32145 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:25:55 dist-test-slave-zbnc krb5kdc[31522](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713555, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.53.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:25:55.532495 32145 init.cc:377] Logged in from keytab as kudu/127.17.53.65@KRBTEST.COM (short username kudu)
I20260929 20:25:55.534067 32145 webserver.cc:460] Webserver started at http://127.17.53.65:46229/ using document root <none> and password file <none>
I20260929 20:25:55.534781 32145 fs_manager.cc:359] Metadata directory not provided
I20260929 20:25:55.534963 32145 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:25:55.540925 32145 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 20:25:55.543944 32181 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:25:55.545492 32145 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 20:25:55.545677 32145 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713339161492-17621-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713339161492-17621-0/minicluster-data/ts-0/wal
uuid: "ad1b6c08928b446ca0b8539b56223f57"
format_stamp: "Formatted at 2026-09-29 20:25:52 on dist-test-slave-zbnc"
server_key: "ea98d3192c7fc95189919bf5c74d03ae"
server_key_iv: "03dc1c8d9834e63fb29a049126bdefe3"
server_key_version: "encryptionkey@0"
I20260929 20:25:55.546687 32145 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713339161492-17621-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713339161492-17621-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713339161492-17621-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:25:55.572394 32145 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:25:55.576781 32145 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:25:55.577080 32145 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:25:55.578231 32145 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:25:55.579768 32145 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:25:55.579912 32145 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:25:55.580042 32145 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:25:55.580132 32145 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:25:55.614566 32145 rpc_server.cc:307] RPC server started. Bound to: 127.17.53.65:36129
I20260929 20:25:55.614650 32293 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.53.65:36129 every 8 connection(s)
I20260929 20:25:55.616129 32145 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713339161492-17621-0/minicluster-data/ts-0/data/info.pb
I20260929 20:25:55.619062 17621 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu as pid 32145
I20260929 20:25:55.619839 17621 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
/tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713339161492-17621-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713339161492-17621-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713339161492-17621-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713339161492-17621-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.53.66:43527
--local_ip_for_outbound_sockets=127.17.53.66
--tserver_master_addrs=127.17.53.126:37213
--webserver_port=45857
--webserver_interface=127.17.53.66
--keytab_file=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.66.keytab
--principal=kudu/127.17.53.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.53.84:41567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713339161492-17621-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713339161492-17621-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713339161492-17621-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 20:25:55 dist-test-slave-zbnc krb5kdc[31522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713555, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.53.65@KRBTEST.COM for kudu/127.17.53.126@KRBTEST.COM
I20260929 20:25:55.642275 32296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:25:55.619140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.53.65:60879 (local address 127.17.53.126:37213)
0929 20:25:55.619802 (+ 662us) server_negotiation.cc:206] Beginning negotiation
0929 20:25:55.619820 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0929 20:25:55.620371 (+ 551us) server_negotiation.cc:407] Connection header received
0929 20:25:55.622470 (+ 2099us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:25:55.622490 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:25:55.622969 (+ 479us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:25:55.623369 (+ 400us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:25:55.624742 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:25:55.626769 (+ 2027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:25:55.629324 (+ 2555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:25:55.629964 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:25:55.630088 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:25:55.634493 (+ 4405us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:25:55.634500 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:25:55.634515 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:25:55.634551 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:25:55.637657 (+ 3106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:25:55.638677 (+ 1020us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:25:55.638683 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:25:55.638689 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:25:55.638902 (+ 213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:25:55.639437 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:25:55.639440 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:25:55.639442 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:25:55.639899 (+ 457us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:25:55.640039 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0929 20:25:55.641038 (+ 999us) server_negotiation.cc:299] Negotiation successful
0929 20:25:55.641355 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":388,"thread_start_us":251,"threads_started":1}
I20260929 20:25:55.642952 32297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:25:55.619318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:37213 (local address 127.17.53.65:60879)
0929 20:25:55.620178 (+ 860us) negotiation.cc:107] Waiting for socket to connect
0929 20:25:55.620224 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0929 20:25:55.621922 (+ 1698us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:25:55.623482 (+ 1560us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:25:55.623489 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:25:55.623962 (+ 473us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:25:55.624506 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:25:55.624522 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:25:55.626969 (+ 2447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:25:55.626976 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:25:55.629184 (+ 2208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:25:55.629201 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:25:55.630199 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:25:55.630208 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:25:55.630514 (+ 306us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:25:55.631261 (+ 747us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:25:55.631293 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:25:55.634223 (+ 2930us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:25:55.637809 (+ 3586us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:25:55.637822 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:25:55.637841 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:25:55.638542 (+ 701us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:25:55.639041 (+ 499us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:25:55.639049 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:25:55.639054 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:25:55.639323 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:25:55.640052 (+ 729us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:25:55.640062 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:25:55.640540 (+ 478us) client_negotiation.cc:769] Sending connection context
0929 20:25:55.640867 (+ 327us) client_negotiation.cc:240] Negotiation successful
0929 20:25:55.641495 (+ 628us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":561,"thread_start_us":341,"threads_started":1}
I20260929 20:25:55.646202 32294 heartbeater.cc:347] Connected to a master server at 127.17.53.126:37213
I20260929 20:25:55.646538 32294 heartbeater.cc:464] Registering TS with master...
I20260929 20:25:55.647363 32294 heartbeater.cc:511] Master 127.17.53.126:37213 requested a full tablet report, sending...
I20260929 20:25:55.649710 32108 ts_manager.cc:194] Registered new tserver with Master: ad1b6c08928b446ca0b8539b56223f57 (127.17.53.65:36129)
I20260929 20:25:55.651873 32108 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.53.65@KRBTEST.COM'} at 127.17.53.65:60879
W20260929 20:25:55.952668 32298 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:25:55.952955 32298 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:25:55.953051 32298 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:25:55.960610 32298 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:25:55.960724 32298 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:25:55.960836 32298 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.53.66
I20260929 20:25:55.970436 32298 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.53.84:41567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713339161492-17621-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713339161492-17621-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.66.keytab
--principal=kudu/127.17.53.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.53.66:43527
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713339161492-17621-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.17.53.66
--webserver_port=45857
--tserver_master_addrs=127.17.53.126:37213
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.53.66
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713339161492-17621-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:25:55.971680 32298 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:25:55.972959 32298 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:25:55.985868 32303 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:25:55.985934 32304 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:25:55.986213 32306 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:25:55.987200 32298 server_base.cc:1034] running on GCE node
I20260929 20:25:55.987766 32298 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:25:55.988651 32298 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:25:55.989861 32298 hybrid_clock.cc:648] HybridClock initialized: now 1790713555989823 us; error 38 us; skew 500 ppm
I20260929 20:25:55.990120 32298 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:25:55 dist-test-slave-zbnc krb5kdc[31522](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713555, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.53.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:25:55.993239 32298 init.cc:377] Logged in from keytab as kudu/127.17.53.66@KRBTEST.COM (short username kudu)
I20260929 20:25:55.994769 32298 webserver.cc:460] Webserver started at http://127.17.53.66:45857/ using document root <none> and password file <none>
I20260929 20:25:55.995478 32298 fs_manager.cc:359] Metadata directory not provided
I20260929 20:25:55.995630 32298 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:25:56.000923 32298 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260929 20:25:56.003762 32313 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:25:56.005406 32298 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 20:25:56.005589 32298 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713339161492-17621-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713339161492-17621-0/minicluster-data/ts-1/wal
uuid: "eedca80ed76e4368a624213c9faed862"
format_stamp: "Formatted at 2026-09-29 20:25:53 on dist-test-slave-zbnc"
server_key: "7b18e8c3cf0d8876f9869e66f8cc41a4"
server_key_iv: "5a1d701cc0dad48f1c12dd09c3debb72"
server_key_version: "encryptionkey@0"
I20260929 20:25:56.006578 32298 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713339161492-17621-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713339161492-17621-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713339161492-17621-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:25:56.030802 32298 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:25:56.035041 32298 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:25:56.035339 32298 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:25:56.036338 32298 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:25:56.037695 32298 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:25:56.037794 32298 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:25:56.037952 32298 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:25:56.038045 32298 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:25:56.070679 32298 rpc_server.cc:307] RPC server started. Bound to: 127.17.53.66:43527
I20260929 20:25:56.070825 32425 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.53.66:43527 every 8 connection(s)
I20260929 20:25:56.072181 32298 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713339161492-17621-0/minicluster-data/ts-1/data/info.pb
I20260929 20:25:56.075736 17621 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu as pid 32298
I20260929 20:25:56.076486 17621 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
/tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713339161492-17621-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713339161492-17621-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713339161492-17621-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713339161492-17621-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.53.67:39129
--local_ip_for_outbound_sockets=127.17.53.67
--tserver_master_addrs=127.17.53.126:37213
--webserver_port=40081
--webserver_interface=127.17.53.67
--keytab_file=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.67.keytab
--principal=kudu/127.17.53.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.53.84:41567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713339161492-17621-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713339161492-17621-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713339161492-17621-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 20:25:56 dist-test-slave-zbnc krb5kdc[31522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713555, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.53.66@KRBTEST.COM for kudu/127.17.53.126@KRBTEST.COM
I20260929 20:25:56.097211 32296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:25:56.074723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.53.66:33691 (local address 127.17.53.126:37213)
0929 20:25:56.074936 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0929 20:25:56.074941 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:25:56.076260 (+ 1319us) server_negotiation.cc:407] Connection header received
0929 20:25:56.079301 (+ 3041us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:25:56.079305 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:25:56.079378 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:25:56.079469 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:25:56.081599 (+ 2130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:25:56.082736 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:25:56.085363 (+ 2627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:25:56.086129 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:25:56.086275 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:25:56.089918 (+ 3643us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:25:56.089927 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:25:56.089935 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:25:56.089982 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:25:56.093643 (+ 3661us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:25:56.094406 (+ 763us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:25:56.094415 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:25:56.094421 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:25:56.094588 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:25:56.095067 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:25:56.095076 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:25:56.095082 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:25:56.095488 (+ 406us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:25:56.095648 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0929 20:25:56.096451 (+ 803us) server_negotiation.cc:299] Negotiation successful
0929 20:25:56.096748 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 20:25:56.097854 32428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:25:56.075225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:37213 (local address 127.17.53.66:33691)
0929 20:25:56.076056 (+ 831us) negotiation.cc:107] Waiting for socket to connect
0929 20:25:56.076113 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0929 20:25:56.078978 (+ 2865us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:25:56.079758 (+ 780us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:25:56.079773 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:25:56.080539 (+ 766us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:25:56.081374 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:25:56.081403 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:25:56.082984 (+ 1581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:25:56.082992 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:25:56.085159 (+ 2167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:25:56.085179 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:25:56.086350 (+ 1171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:25:56.086355 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:25:56.086574 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:25:56.087298 (+ 724us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:25:56.087321 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:25:56.089706 (+ 2385us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:25:56.093796 (+ 4090us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:25:56.093803 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:25:56.093817 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:25:56.094264 (+ 447us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:25:56.094698 (+ 434us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:25:56.094701 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:25:56.094703 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:25:56.094946 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:25:56.095648 (+ 702us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:25:56.095653 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:25:56.095986 (+ 333us) client_negotiation.cc:769] Sending connection context
0929 20:25:56.096282 (+ 296us) client_negotiation.cc:240] Negotiation successful
0929 20:25:56.096718 (+ 436us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":532,"thread_start_us":311,"threads_started":1}
I20260929 20:25:56.099362 32426 heartbeater.cc:347] Connected to a master server at 127.17.53.126:37213
I20260929 20:25:56.099761 32426 heartbeater.cc:464] Registering TS with master...
I20260929 20:25:56.100816 32426 heartbeater.cc:511] Master 127.17.53.126:37213 requested a full tablet report, sending...
I20260929 20:25:56.102553 32108 ts_manager.cc:194] Registered new tserver with Master: eedca80ed76e4368a624213c9faed862 (127.17.53.66:43527)
I20260929 20:25:56.103451 32108 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.53.66@KRBTEST.COM'} at 127.17.53.66:33691
W20260929 20:25:56.418291 32429 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:25:56.418581 32429 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:25:56.418752 32429 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:25:56.425982 32429 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:25:56.426097 32429 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:25:56.426359 32429 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.53.67
I20260929 20:25:56.436100 32429 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.53.84:41567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713339161492-17621-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713339161492-17621-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.67.keytab
--principal=kudu/127.17.53.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.53.67:39129
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713339161492-17621-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.17.53.67
--webserver_port=40081
--tserver_master_addrs=127.17.53.126:37213
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.53.67
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713339161492-17621-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:25:56.437422 32429 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:25:56.438755 32429 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:25:56.451784 32437 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:25:56.451810 32435 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:25:56.451959 32434 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:25:56.452708 32429 server_base.cc:1034] running on GCE node
I20260929 20:25:56.453420 32429 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:25:56.454259 32429 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:25:56.455498 32429 hybrid_clock.cc:648] HybridClock initialized: now 1790713556455448 us; error 43 us; skew 500 ppm
I20260929 20:25:56.455807 32429 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:25:56 dist-test-slave-zbnc krb5kdc[31522](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713556, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.53.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:25:56.458788 32429 init.cc:377] Logged in from keytab as kudu/127.17.53.67@KRBTEST.COM (short username kudu)
I20260929 20:25:56.460294 32429 webserver.cc:460] Webserver started at http://127.17.53.67:40081/ using document root <none> and password file <none>
I20260929 20:25:56.460960 32429 fs_manager.cc:359] Metadata directory not provided
I20260929 20:25:56.461119 32429 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:25:56.466470 32429 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.002s
I20260929 20:25:56.469249 32444 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:25:56.470641 32429 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260929 20:25:56.470887 32429 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713339161492-17621-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713339161492-17621-0/minicluster-data/ts-2/wal
uuid: "77871fa252864a14a60aefdf7b95e58d"
format_stamp: "Formatted at 2026-09-29 20:25:53 on dist-test-slave-zbnc"
server_key: "283af3aeaffef06b486bfbab35cd9520"
server_key_iv: "92e1a9b45c86a47eae578b5751b56ce6"
server_key_version: "encryptionkey@0"
I20260929 20:25:56.471928 32429 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713339161492-17621-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713339161492-17621-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713339161492-17621-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:25:56.498018 32429 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:25:56.502501 32429 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:25:56.502789 32429 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:25:56.503891 32429 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:25:56.505333 32429 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:25:56.505457 32429 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:25:56.505606 32429 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:25:56.505704 32429 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:25:56.539115 32429 rpc_server.cc:307] RPC server started. Bound to: 127.17.53.67:39129
I20260929 20:25:56.539206 32556 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.53.67:39129 every 8 connection(s)
I20260929 20:25:56.540630 32429 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713339161492-17621-0/minicluster-data/ts-2/data/info.pb
I20260929 20:25:56.544842 17621 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu as pid 32429
Sep 29 20:25:56 dist-test-slave-zbnc krb5kdc[31522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713556, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.53.67@KRBTEST.COM for kudu/127.17.53.126@KRBTEST.COM
I20260929 20:25:56.562006 32561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:25:56.545827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53022 (local address 127.17.53.126:37213)
0929 20:25:56.547227 (+ 1400us) server_negotiation.cc:206] Beginning negotiation
0929 20:25:56.547231 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:25:56.547246 (+ 15us) server_negotiation.cc:407] Connection header received
0929 20:25:56.547292 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:25:56.547295 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:25:56.547355 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:25:56.547447 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:25:56.548999 (+ 1552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:25:56.549894 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:25:56.551490 (+ 1596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:25:56.552020 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:25:56.552110 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:25:56.555194 (+ 3084us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:25:56.555199 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:25:56.555203 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:25:56.555240 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:25:56.558947 (+ 3707us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:25:56.559741 (+ 794us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:25:56.559751 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:25:56.559756 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:25:56.559908 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:25:56.560364 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:25:56.560370 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:25:56.560374 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:25:56.560670 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:25:56.560951 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0929 20:25:56.561275 (+ 324us) server_negotiation.cc:299] Negotiation successful
0929 20:25:56.561522 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":6,"server-negotiator.queue_time_us":1287,"thread_start_us":1207,"threads_started":1}
I20260929 20:25:56.562393 32296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:25:56.543274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.53.67:35317 (local address 127.17.53.126:37213)
0929 20:25:56.543495 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0929 20:25:56.543503 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 20:25:56.544843 (+ 1340us) server_negotiation.cc:407] Connection header received
0929 20:25:56.547283 (+ 2440us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:25:56.547289 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:25:56.547391 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:25:56.547508 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:25:56.549064 (+ 1556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:25:56.549905 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:25:56.551635 (+ 1730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:25:56.552185 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:25:56.552319 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:25:56.556076 (+ 3757us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:25:56.556084 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:25:56.556088 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:25:56.556125 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:25:56.559400 (+ 3275us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:25:56.560085 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:25:56.560091 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:25:56.560094 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:25:56.560232 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:25:56.560658 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:25:56.560663 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:25:56.560667 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:25:56.560949 (+ 282us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:25:56.561096 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0929 20:25:56.561798 (+ 702us) server_negotiation.cc:299] Negotiation successful
0929 20:25:56.562000 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":84,"server-negotiator.queue_time_us":77,"spinlock_wait_cycles":37120}
I20260929 20:25:56.563490 32559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:25:56.543847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:37213 (local address 127.17.53.67:35317)
0929 20:25:56.544626 (+ 779us) negotiation.cc:107] Waiting for socket to connect
0929 20:25:56.544687 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0929 20:25:56.547013 (+ 2326us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:25:56.547794 (+ 781us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:25:56.547803 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:25:56.548299 (+ 496us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:25:56.548869 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:25:56.548886 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:25:56.550082 (+ 1196us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:25:56.550085 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:25:56.551505 (+ 1420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:25:56.551519 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:25:56.552395 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:25:56.552399 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:25:56.552594 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:25:56.553280 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:25:56.553301 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:25:56.555803 (+ 2502us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:25:56.559524 (+ 3721us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:25:56.559533 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:25:56.559550 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:25:56.559959 (+ 409us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:25:56.560345 (+ 386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:25:56.560350 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:25:56.560353 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:25:56.560537 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:25:56.561086 (+ 549us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:25:56.561094 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:25:56.561431 (+ 337us) client_negotiation.cc:769] Sending connection context
0929 20:25:56.561693 (+ 262us) client_negotiation.cc:240] Negotiation successful
0929 20:25:56.562241 (+ 548us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":259,"threads_started":1}
I20260929 20:25:56.564870 32557 heartbeater.cc:347] Connected to a master server at 127.17.53.126:37213
I20260929 20:25:56.565222 32557 heartbeater.cc:464] Registering TS with master...
I20260929 20:25:56.565932 32557 heartbeater.cc:511] Master 127.17.53.126:37213 requested a full tablet report, sending...
I20260929 20:25:56.567332 32108 ts_manager.cc:194] Registered new tserver with Master: 77871fa252864a14a60aefdf7b95e58d (127.17.53.67:39129)
I20260929 20:25:56.568147 32108 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.53.67@KRBTEST.COM'} at 127.17.53.67:35317
I20260929 20:25:56.577927 17621 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 20:25:56.585579 32296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:25:56.579056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53026 (local address 127.17.53.126:37213)
0929 20:25:56.579278 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0929 20:25:56.579287 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 20:25:56.579707 (+ 420us) server_negotiation.cc:407] Connection header received
0929 20:25:56.579889 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:25:56.579893 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:25:56.580076 (+ 183us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:25:56.580196 (+ 120us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 20:25:56.580678 (+ 482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:25:56.581395 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:25:56.583183 (+ 1788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:25:56.583658 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:25:56.583787 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:25:56.584312 (+ 525us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 20:25:56.584836 (+ 524us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 20:25:56.584959 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0929 20:25:56.585183 (+ 224us) server_negotiation.cc:299] Negotiation successful
0929 20:25:56.585256 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260929 20:25:56.586522 32108 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:53026:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 20:25:56.594668 32108 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 20:25:56.629528 32570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:25:56.608367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58312 (local address 127.17.53.66:43527)
0929 20:25:56.620446 (+ 12079us) server_negotiation.cc:206] Beginning negotiation
0929 20:25:56.620459 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 20:25:56.620497 (+ 38us) server_negotiation.cc:407] Connection header received
0929 20:25:56.620629 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:25:56.620640 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:25:56.621940 (+ 1300us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:25:56.622226 (+ 286us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:25:56.622948 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:25:56.624457 (+ 1509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:25:56.627026 (+ 2569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:25:56.628338 (+ 1312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:25:56.628489 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:25:56.628679 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0929 20:25:56.629013 (+ 334us) server_negotiation.cc:299] Negotiation successful
0929 20:25:56.629204 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11867,"thread_start_us":11853,"threads_started":1}
I20260929 20:25:56.629609 32567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:25:56.607860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.66:43527 (local address 127.0.0.1:58312)
0929 20:25:56.608645 (+ 785us) negotiation.cc:107] Waiting for socket to connect
0929 20:25:56.608708 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0929 20:25:56.608867 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:25:56.622225 (+ 13358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:25:56.622235 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:25:56.622272 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:25:56.622746 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:25:56.622763 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:25:56.624638 (+ 1875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:25:56.624644 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:25:56.626820 (+ 2176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:25:56.626832 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:25:56.628557 (+ 1725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:25:56.628563 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:25:56.628750 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:25:56.628804 (+ 54us) client_negotiation.cc:769] Sending connection context
0929 20:25:56.628937 (+ 133us) client_negotiation.cc:240] Negotiation successful
0929 20:25:56.629129 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":522,"mutex_wait_us":129,"thread_start_us":709,"threads_started":1}
I20260929 20:25:56.630255 32569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:25:56.607926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.65:36129 (local address 127.0.0.1:39620)
0929 20:25:56.619653 (+ 11727us) negotiation.cc:107] Waiting for socket to connect
0929 20:25:56.619692 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0929 20:25:56.619859 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:25:56.622563 (+ 2704us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:25:56.622570 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:25:56.622593 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:25:56.623003 (+ 410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:25:56.623014 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:25:56.624852 (+ 1838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:25:56.624860 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:25:56.627283 (+ 2423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:25:56.627298 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:25:56.629369 (+ 2071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:25:56.629375 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:25:56.629561 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:25:56.629605 (+ 44us) client_negotiation.cc:769] Sending connection context
0929 20:25:56.629698 (+ 93us) client_negotiation.cc:240] Negotiation successful
0929 20:25:56.629843 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11531,"mutex_wait_us":127,"thread_start_us":12534,"threads_started":1}
I20260929 20:25:56.631194 32571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:25:56.620842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.67:39129 (local address 127.0.0.1:54642)
0929 20:25:56.621427 (+ 585us) negotiation.cc:107] Waiting for socket to connect
0929 20:25:56.621452 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0929 20:25:56.621570 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:25:56.623521 (+ 1951us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:25:56.623526 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:25:56.623551 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:25:56.623888 (+ 337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:25:56.623899 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:25:56.625726 (+ 1827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:25:56.625734 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:25:56.627827 (+ 2093us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:25:56.627837 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:25:56.630136 (+ 2299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:25:56.630148 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:25:56.630582 (+ 434us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:25:56.630613 (+ 31us) client_negotiation.cc:769] Sending connection context
0929 20:25:56.630713 (+ 100us) client_negotiation.cc:240] Negotiation successful
0929 20:25:56.630797 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":449,"mutex_wait_us":2,"thread_start_us":306,"threads_started":1}
I20260929 20:25:56.631321 32568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:25:56.607904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39620 (local address 127.17.53.65:36129)
0929 20:25:56.621836 (+ 13932us) server_negotiation.cc:206] Beginning negotiation
0929 20:25:56.621842 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:25:56.621863 (+ 21us) server_negotiation.cc:407] Connection header received
0929 20:25:56.621939 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:25:56.621944 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:25:56.622105 (+ 161us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:25:56.622332 (+ 227us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:25:56.623177 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:25:56.624666 (+ 1489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:25:56.627441 (+ 2775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:25:56.628852 (+ 1411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:25:56.629003 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:25:56.629206 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0929 20:25:56.630764 (+ 1558us) server_negotiation.cc:299] Negotiation successful
0929 20:25:56.630946 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13798,"thread_start_us":13662,"threads_started":1}
I20260929 20:25:56.631552 32572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:25:56.621232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54642 (local address 127.17.53.67:39129)
0929 20:25:56.622963 (+ 1731us) server_negotiation.cc:206] Beginning negotiation
0929 20:25:56.622971 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 20:25:56.623001 (+ 30us) server_negotiation.cc:407] Connection header received
0929 20:25:56.623098 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:25:56.623104 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:25:56.623332 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:25:56.623511 (+ 179us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:25:56.624062 (+ 551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:25:56.625586 (+ 1524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:25:56.627967 (+ 2381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:25:56.629178 (+ 1211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:25:56.629303 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:25:56.629508 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0929 20:25:56.630976 (+ 1468us) server_negotiation.cc:299] Negotiation successful
0929 20:25:56.631148 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1544,"thread_start_us":1381,"threads_started":1}
I20260929 20:25:56.633977 32361 tablet_service.cc:1467] Processing CreateTablet for tablet 127e6bec99db4dba8ba6ae51ea11e36d (DEFAULT_TABLE table=test-table [id=67161f25a1ca465ab65cf77a7fe59450]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:25:56.634594 32229 tablet_service.cc:1467] Processing CreateTablet for tablet 127e6bec99db4dba8ba6ae51ea11e36d (DEFAULT_TABLE table=test-table [id=67161f25a1ca465ab65cf77a7fe59450]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:25:56.635289 32361 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 127e6bec99db4dba8ba6ae51ea11e36d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:25:56.635972 32229 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 127e6bec99db4dba8ba6ae51ea11e36d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:25:56.635433 32492 tablet_service.cc:1467] Processing CreateTablet for tablet 127e6bec99db4dba8ba6ae51ea11e36d (DEFAULT_TABLE table=test-table [id=67161f25a1ca465ab65cf77a7fe59450]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:25:56.636745 32492 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 127e6bec99db4dba8ba6ae51ea11e36d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:25:56.647432 32573 tablet_bootstrap.cc:492] T 127e6bec99db4dba8ba6ae51ea11e36d P eedca80ed76e4368a624213c9faed862: Bootstrap starting.
I20260929 20:25:56.651638 32573 tablet_bootstrap.cc:654] T 127e6bec99db4dba8ba6ae51ea11e36d P eedca80ed76e4368a624213c9faed862: Neither blocks nor log segments found. Creating new log.
I20260929 20:25:56.651820 32575 tablet_bootstrap.cc:492] T 127e6bec99db4dba8ba6ae51ea11e36d P 77871fa252864a14a60aefdf7b95e58d: Bootstrap starting.
I20260929 20:25:56.652693 32574 tablet_bootstrap.cc:492] T 127e6bec99db4dba8ba6ae51ea11e36d P ad1b6c08928b446ca0b8539b56223f57: Bootstrap starting.
I20260929 20:25:56.653360 32573 log.cc:824] T 127e6bec99db4dba8ba6ae51ea11e36d P eedca80ed76e4368a624213c9faed862: Log is configured to *not* fsync() on all Append() calls
I20260929 20:25:56.655117 32294 heartbeater.cc:503] Master 127.17.53.126:37213 was elected leader, sending a full tablet report...
I20260929 20:25:56.655309 32575 tablet_bootstrap.cc:654] T 127e6bec99db4dba8ba6ae51ea11e36d P 77871fa252864a14a60aefdf7b95e58d: Neither blocks nor log segments found. Creating new log.
I20260929 20:25:56.655944 32574 tablet_bootstrap.cc:654] T 127e6bec99db4dba8ba6ae51ea11e36d P ad1b6c08928b446ca0b8539b56223f57: Neither blocks nor log segments found. Creating new log.
I20260929 20:25:56.656522 32575 log.cc:824] T 127e6bec99db4dba8ba6ae51ea11e36d P 77871fa252864a14a60aefdf7b95e58d: Log is configured to *not* fsync() on all Append() calls
I20260929 20:25:56.656622 32573 tablet_bootstrap.cc:492] T 127e6bec99db4dba8ba6ae51ea11e36d P eedca80ed76e4368a624213c9faed862: No bootstrap required, opened a new log
I20260929 20:25:56.656874 32573 ts_tablet_manager.cc:1397] T 127e6bec99db4dba8ba6ae51ea11e36d P eedca80ed76e4368a624213c9faed862: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.006s
I20260929 20:25:56.657121 32574 log.cc:824] T 127e6bec99db4dba8ba6ae51ea11e36d P ad1b6c08928b446ca0b8539b56223f57: Log is configured to *not* fsync() on all Append() calls
I20260929 20:25:56.659911 32574 tablet_bootstrap.cc:492] T 127e6bec99db4dba8ba6ae51ea11e36d P ad1b6c08928b446ca0b8539b56223f57: No bootstrap required, opened a new log
I20260929 20:25:56.660228 32574 ts_tablet_manager.cc:1397] T 127e6bec99db4dba8ba6ae51ea11e36d P ad1b6c08928b446ca0b8539b56223f57: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260929 20:25:56.665647 32575 tablet_bootstrap.cc:492] T 127e6bec99db4dba8ba6ae51ea11e36d P 77871fa252864a14a60aefdf7b95e58d: No bootstrap required, opened a new log
I20260929 20:25:56.665632 32573 raft_consensus.cc:348] T 127e6bec99db4dba8ba6ae51ea11e36d P eedca80ed76e4368a624213c9faed862 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad1b6c08928b446ca0b8539b56223f57" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 36129 } } peers { permanent_uuid: "eedca80ed76e4368a624213c9faed862" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 43527 } } peers { permanent_uuid: "77871fa252864a14a60aefdf7b95e58d" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 39129 } }
I20260929 20:25:56.665966 32575 ts_tablet_manager.cc:1397] T 127e6bec99db4dba8ba6ae51ea11e36d P 77871fa252864a14a60aefdf7b95e58d: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.006s
I20260929 20:25:56.666090 32573 raft_consensus.cc:374] T 127e6bec99db4dba8ba6ae51ea11e36d P eedca80ed76e4368a624213c9faed862 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:25:56.666222 32573 raft_consensus.cc:729] T 127e6bec99db4dba8ba6ae51ea11e36d P eedca80ed76e4368a624213c9faed862 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eedca80ed76e4368a624213c9faed862, State: Initialized, Role: FOLLOWER
I20260929 20:25:56.666913 32573 consensus_queue.cc:260] T 127e6bec99db4dba8ba6ae51ea11e36d P eedca80ed76e4368a624213c9faed862 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad1b6c08928b446ca0b8539b56223f57" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 36129 } } peers { permanent_uuid: "eedca80ed76e4368a624213c9faed862" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 43527 } } peers { permanent_uuid: "77871fa252864a14a60aefdf7b95e58d" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 39129 } }
I20260929 20:25:56.668076 32426 heartbeater.cc:503] Master 127.17.53.126:37213 was elected leader, sending a full tablet report...
I20260929 20:25:56.668707 32573 ts_tablet_manager.cc:1428] T 127e6bec99db4dba8ba6ae51ea11e36d P eedca80ed76e4368a624213c9faed862: Time spent starting tablet: real 0.012s user 0.006s sys 0.006s
I20260929 20:25:56.672330 32574 raft_consensus.cc:348] T 127e6bec99db4dba8ba6ae51ea11e36d P ad1b6c08928b446ca0b8539b56223f57 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad1b6c08928b446ca0b8539b56223f57" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 36129 } } peers { permanent_uuid: "eedca80ed76e4368a624213c9faed862" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 43527 } } peers { permanent_uuid: "77871fa252864a14a60aefdf7b95e58d" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 39129 } }
I20260929 20:25:56.672843 32574 raft_consensus.cc:374] T 127e6bec99db4dba8ba6ae51ea11e36d P ad1b6c08928b446ca0b8539b56223f57 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:25:56.672952 32574 raft_consensus.cc:729] T 127e6bec99db4dba8ba6ae51ea11e36d P ad1b6c08928b446ca0b8539b56223f57 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad1b6c08928b446ca0b8539b56223f57, State: Initialized, Role: FOLLOWER
I20260929 20:25:56.673615 32574 consensus_queue.cc:260] T 127e6bec99db4dba8ba6ae51ea11e36d P ad1b6c08928b446ca0b8539b56223f57 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad1b6c08928b446ca0b8539b56223f57" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 36129 } } peers { permanent_uuid: "eedca80ed76e4368a624213c9faed862" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 43527 } } peers { permanent_uuid: "77871fa252864a14a60aefdf7b95e58d" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 39129 } }
I20260929 20:25:56.675606 32574 ts_tablet_manager.cc:1428] T 127e6bec99db4dba8ba6ae51ea11e36d P ad1b6c08928b446ca0b8539b56223f57: Time spent starting tablet: real 0.015s user 0.014s sys 0.001s
I20260929 20:25:56.675971 32575 raft_consensus.cc:348] T 127e6bec99db4dba8ba6ae51ea11e36d P 77871fa252864a14a60aefdf7b95e58d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad1b6c08928b446ca0b8539b56223f57" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 36129 } } peers { permanent_uuid: "eedca80ed76e4368a624213c9faed862" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 43527 } } peers { permanent_uuid: "77871fa252864a14a60aefdf7b95e58d" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 39129 } }
I20260929 20:25:56.676385 32575 raft_consensus.cc:374] T 127e6bec99db4dba8ba6ae51ea11e36d P 77871fa252864a14a60aefdf7b95e58d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:25:56.676507 32575 raft_consensus.cc:729] T 127e6bec99db4dba8ba6ae51ea11e36d P 77871fa252864a14a60aefdf7b95e58d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77871fa252864a14a60aefdf7b95e58d, State: Initialized, Role: FOLLOWER
I20260929 20:25:56.677175 32575 consensus_queue.cc:260] T 127e6bec99db4dba8ba6ae51ea11e36d P 77871fa252864a14a60aefdf7b95e58d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad1b6c08928b446ca0b8539b56223f57" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 36129 } } peers { permanent_uuid: "eedca80ed76e4368a624213c9faed862" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 43527 } } peers { permanent_uuid: "77871fa252864a14a60aefdf7b95e58d" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 39129 } }
I20260929 20:25:56.678359 32557 heartbeater.cc:503] Master 127.17.53.126:37213 was elected leader, sending a full tablet report...
I20260929 20:25:56.679108 32575 ts_tablet_manager.cc:1428] T 127e6bec99db4dba8ba6ae51ea11e36d P 77871fa252864a14a60aefdf7b95e58d: Time spent starting tablet: real 0.013s user 0.014s sys 0.002s
W20260929 20:25:56.792346 32558 tablet.cc:2369] T 127e6bec99db4dba8ba6ae51ea11e36d P 77871fa252864a14a60aefdf7b95e58d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 20:25:56.794693 32581 raft_consensus.cc:482] T 127e6bec99db4dba8ba6ae51ea11e36d P 77871fa252864a14a60aefdf7b95e58d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 20:25:56.795039 32581 raft_consensus.cc:504] T 127e6bec99db4dba8ba6ae51ea11e36d P 77871fa252864a14a60aefdf7b95e58d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad1b6c08928b446ca0b8539b56223f57" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 36129 } } peers { permanent_uuid: "eedca80ed76e4368a624213c9faed862" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 43527 } } peers { permanent_uuid: "77871fa252864a14a60aefdf7b95e58d" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 39129 } }
I20260929 20:25:56.796828 32581 leader_election.cc:288] T 127e6bec99db4dba8ba6ae51ea11e36d P 77871fa252864a14a60aefdf7b95e58d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ad1b6c08928b446ca0b8539b56223f57 (127.17.53.65:36129), eedca80ed76e4368a624213c9faed862 (127.17.53.66:43527)
I20260929 20:25:56.803469 32559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:25:56.797069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.65:36129 (local address 127.17.53.67:51569)
0929 20:25:56.797300 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0929 20:25:56.797324 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0929 20:25:56.797473 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:25:56.798012 (+ 539us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:25:56.798017 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:25:56.798048 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:25:56.798290 (+ 242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:25:56.798303 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:25:56.799868 (+ 1565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:25:56.799873 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:25:56.801647 (+ 1774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:25:56.801659 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:25:56.802718 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:25:56.802726 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:25:56.802930 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:25:56.802955 (+ 25us) client_negotiation.cc:769] Sending connection context
0929 20:25:56.803055 (+ 100us) client_negotiation.cc:240] Negotiation successful
0929 20:25:56.803183 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37,"mutex_wait_us":63}
I20260929 20:25:56.803469 32568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:25:56.797281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.53.67:51569 (local address 127.17.53.65:36129)
0929 20:25:56.797500 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0929 20:25:56.797505 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:25:56.797562 (+ 57us) server_negotiation.cc:407] Connection header received
0929 20:25:56.797649 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:25:56.797658 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:25:56.797778 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:25:56.797874 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:25:56.798488 (+ 614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:25:56.799702 (+ 1214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:25:56.801800 (+ 2098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:25:56.802564 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:25:56.802750 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:25:56.802830 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0929 20:25:56.803093 (+ 263us) server_negotiation.cc:299] Negotiation successful
0929 20:25:56.803225 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260929 20:25:56.804507 32582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:25:56.797372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.66:43527 (local address 127.17.53.67:51847)
0929 20:25:56.798117 (+ 745us) negotiation.cc:107] Waiting for socket to connect
0929 20:25:56.798133 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 20:25:56.798256 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:25:56.798748 (+ 492us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:25:56.798751 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:25:56.798770 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:25:56.799077 (+ 307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:25:56.799085 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:25:56.800506 (+ 1421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:25:56.800510 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:25:56.802367 (+ 1857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:25:56.802381 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:25:56.803689 (+ 1308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:25:56.803696 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:25:56.803879 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:25:56.803904 (+ 25us) client_negotiation.cc:769] Sending connection context
0929 20:25:56.804013 (+ 109us) client_negotiation.cc:240] Negotiation successful
0929 20:25:56.804145 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":628,"thread_start_us":411,"threads_started":1}
I20260929 20:25:56.804522 32570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:25:56.797696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.53.67:51847 (local address 127.17.53.66:43527)
0929 20:25:56.797911 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0929 20:25:56.797919 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 20:25:56.798265 (+ 346us) server_negotiation.cc:407] Connection header received
0929 20:25:56.798489 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:25:56.798495 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:25:56.798609 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:25:56.798732 (+ 123us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:25:56.799232 (+ 500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:25:56.800325 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:25:56.802530 (+ 2205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:25:56.803457 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:25:56.803644 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:25:56.803723 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0929 20:25:56.804050 (+ 327us) server_negotiation.cc:299] Negotiation successful
0929 20:25:56.804252 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260929 20:25:56.805028 32249 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "127e6bec99db4dba8ba6ae51ea11e36d" candidate_uuid: "77871fa252864a14a60aefdf7b95e58d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad1b6c08928b446ca0b8539b56223f57" is_pre_election: true
I20260929 20:25:56.805707 32249 raft_consensus.cc:2454] T 127e6bec99db4dba8ba6ae51ea11e36d P ad1b6c08928b446ca0b8539b56223f57 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77871fa252864a14a60aefdf7b95e58d in term 0.
I20260929 20:25:56.805650 32381 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "127e6bec99db4dba8ba6ae51ea11e36d" candidate_uuid: "77871fa252864a14a60aefdf7b95e58d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eedca80ed76e4368a624213c9faed862" is_pre_election: true
I20260929 20:25:56.806131 32381 raft_consensus.cc:2454] T 127e6bec99db4dba8ba6ae51ea11e36d P eedca80ed76e4368a624213c9faed862 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 77871fa252864a14a60aefdf7b95e58d in term 0.
I20260929 20:25:56.806398 32448 leader_election.cc:302] T 127e6bec99db4dba8ba6ae51ea11e36d P 77871fa252864a14a60aefdf7b95e58d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 77871fa252864a14a60aefdf7b95e58d, ad1b6c08928b446ca0b8539b56223f57; no voters:
I20260929 20:25:56.806836 32581 raft_consensus.cc:2781] T 127e6bec99db4dba8ba6ae51ea11e36d P 77871fa252864a14a60aefdf7b95e58d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 20:25:56.807085 32581 raft_consensus.cc:482] T 127e6bec99db4dba8ba6ae51ea11e36d P 77871fa252864a14a60aefdf7b95e58d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 20:25:56.807214 32581 raft_consensus.cc:3037] T 127e6bec99db4dba8ba6ae51ea11e36d P 77871fa252864a14a60aefdf7b95e58d [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:25:56.809602 32581 raft_consensus.cc:504] T 127e6bec99db4dba8ba6ae51ea11e36d P 77871fa252864a14a60aefdf7b95e58d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad1b6c08928b446ca0b8539b56223f57" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 36129 } } peers { permanent_uuid: "eedca80ed76e4368a624213c9faed862" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 43527 } } peers { permanent_uuid: "77871fa252864a14a60aefdf7b95e58d" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 39129 } }
I20260929 20:25:56.810324 32581 leader_election.cc:288] T 127e6bec99db4dba8ba6ae51ea11e36d P 77871fa252864a14a60aefdf7b95e58d [CANDIDATE]: Term 1 election: Requested vote from peers ad1b6c08928b446ca0b8539b56223f57 (127.17.53.65:36129), eedca80ed76e4368a624213c9faed862 (127.17.53.66:43527)
I20260929 20:25:56.810847 32249 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "127e6bec99db4dba8ba6ae51ea11e36d" candidate_uuid: "77871fa252864a14a60aefdf7b95e58d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ad1b6c08928b446ca0b8539b56223f57"
I20260929 20:25:56.810847 32381 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "127e6bec99db4dba8ba6ae51ea11e36d" candidate_uuid: "77871fa252864a14a60aefdf7b95e58d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eedca80ed76e4368a624213c9faed862"
I20260929 20:25:56.811111 32249 raft_consensus.cc:3037] T 127e6bec99db4dba8ba6ae51ea11e36d P ad1b6c08928b446ca0b8539b56223f57 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:25:56.811213 32381 raft_consensus.cc:3037] T 127e6bec99db4dba8ba6ae51ea11e36d P eedca80ed76e4368a624213c9faed862 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:25:56.814764 32381 raft_consensus.cc:2454] T 127e6bec99db4dba8ba6ae51ea11e36d P eedca80ed76e4368a624213c9faed862 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77871fa252864a14a60aefdf7b95e58d in term 1.
I20260929 20:25:56.814762 32249 raft_consensus.cc:2454] T 127e6bec99db4dba8ba6ae51ea11e36d P ad1b6c08928b446ca0b8539b56223f57 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 77871fa252864a14a60aefdf7b95e58d in term 1.
I20260929 20:25:56.815215 32448 leader_election.cc:302] T 127e6bec99db4dba8ba6ae51ea11e36d P 77871fa252864a14a60aefdf7b95e58d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 77871fa252864a14a60aefdf7b95e58d, ad1b6c08928b446ca0b8539b56223f57, eedca80ed76e4368a624213c9faed862; no voters:
I20260929 20:25:56.815558 32581 raft_consensus.cc:2781] T 127e6bec99db4dba8ba6ae51ea11e36d P 77871fa252864a14a60aefdf7b95e58d [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:25:56.816017 32581 raft_consensus.cc:686] T 127e6bec99db4dba8ba6ae51ea11e36d P 77871fa252864a14a60aefdf7b95e58d [term 1 LEADER]: Becoming Leader. State: Replica: 77871fa252864a14a60aefdf7b95e58d, State: Running, Role: LEADER
I20260929 20:25:56.816618 32581 consensus_queue.cc:237] T 127e6bec99db4dba8ba6ae51ea11e36d P 77871fa252864a14a60aefdf7b95e58d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad1b6c08928b446ca0b8539b56223f57" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 36129 } } peers { permanent_uuid: "eedca80ed76e4368a624213c9faed862" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 43527 } } peers { permanent_uuid: "77871fa252864a14a60aefdf7b95e58d" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 39129 } }
I20260929 20:25:56.821141 32108 catalog_manager.cc:5495] T 127e6bec99db4dba8ba6ae51ea11e36d P 77871fa252864a14a60aefdf7b95e58d reported cstate change: term changed from 0 to 1, leader changed from <none> to 77871fa252864a14a60aefdf7b95e58d (127.17.53.67). New cstate: current_term: 1 leader_uuid: "77871fa252864a14a60aefdf7b95e58d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad1b6c08928b446ca0b8539b56223f57" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 36129 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eedca80ed76e4368a624213c9faed862" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 43527 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "77871fa252864a14a60aefdf7b95e58d" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 39129 } health_report { overall_health: HEALTHY } } }
W20260929 20:25:56.824142 32427 tablet.cc:2369] T 127e6bec99db4dba8ba6ae51ea11e36d P eedca80ed76e4368a624213c9faed862: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 20:25:56.860114 32572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:25:56.854443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54652 (local address 127.17.53.67:39129)
0929 20:25:56.854662 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0929 20:25:56.854668 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:25:56.854772 (+ 104us) server_negotiation.cc:407] Connection header received
0929 20:25:56.855070 (+ 298us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:25:56.855075 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:25:56.855163 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:25:56.855322 (+ 159us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 20:25:56.855689 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:25:56.856429 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:25:56.857950 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:25:56.858354 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:25:56.858492 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:25:56.859028 (+ 536us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 20:25:56.859527 (+ 499us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 20:25:56.859686 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0929 20:25:56.859773 (+ 87us) server_negotiation.cc:299] Negotiation successful
0929 20:25:56.859853 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260929 20:25:56.868228 32381 raft_consensus.cc:1261] T 127e6bec99db4dba8ba6ae51ea11e36d P eedca80ed76e4368a624213c9faed862 [term 1 FOLLOWER]: Refusing update from remote peer 77871fa252864a14a60aefdf7b95e58d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:25:56.868268 32249 raft_consensus.cc:1261] T 127e6bec99db4dba8ba6ae51ea11e36d P ad1b6c08928b446ca0b8539b56223f57 [term 1 FOLLOWER]: Refusing update from remote peer 77871fa252864a14a60aefdf7b95e58d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260929 20:25:56.868494 32295 tablet.cc:2369] T 127e6bec99db4dba8ba6ae51ea11e36d P ad1b6c08928b446ca0b8539b56223f57: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 20:25:56.869326 32583 consensus_queue.cc:1035] T 127e6bec99db4dba8ba6ae51ea11e36d P 77871fa252864a14a60aefdf7b95e58d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ad1b6c08928b446ca0b8539b56223f57" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 36129 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:25:56.869673 32581 consensus_queue.cc:1035] T 127e6bec99db4dba8ba6ae51ea11e36d P 77871fa252864a14a60aefdf7b95e58d [LEADER]: Connected to new peer: Peer: permanent_uuid: "eedca80ed76e4368a624213c9faed862" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 43527 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:25:56.881875 32586 mvcc.cc:205] Tried to move back new op lower bound from 7334762728922419200 to 7334762728724746240. Current Snapshot: MvccSnapshot[applied={T|T < 7334762728922419200}]
I20260929 20:25:56.888669 32587 mvcc.cc:205] Tried to move back new op lower bound from 7334762728922419200 to 7334762728724746240. Current Snapshot: MvccSnapshot[applied={T|T < 7334762728922419200}]
I20260929 20:25:56.897064 32107 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:53026:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 20:25:56.897439 32107 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:53026:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 20:25:56.901517 32107 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7caaae55b00c4da99832e09118f54e5e: Sending DeleteTablet for 3 replicas of tablet 127e6bec99db4dba8ba6ae51ea11e36d
I20260929 20:25:56.902920 32229 tablet_service.cc:1514] Processing DeleteTablet for tablet 127e6bec99db4dba8ba6ae51ea11e36d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:25:56 UTC) from {username='kudu', principal='kudu/127.17.53.126@KRBTEST.COM'} at 127.0.0.1:39620
I20260929 20:25:56.902904 32361 tablet_service.cc:1514] Processing DeleteTablet for tablet 127e6bec99db4dba8ba6ae51ea11e36d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:25:56 UTC) from {username='kudu', principal='kudu/127.17.53.126@KRBTEST.COM'} at 127.0.0.1:58312
I20260929 20:25:56.902945 32492 tablet_service.cc:1514] Processing DeleteTablet for tablet 127e6bec99db4dba8ba6ae51ea11e36d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:25:56 UTC) from {username='kudu', principal='kudu/127.17.53.126@KRBTEST.COM'} at 127.0.0.1:54642
I20260929 20:25:56.904479 17621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu pid 32145
I20260929 20:25:56.939571 32568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:25:56.905534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39630 (local address 127.17.53.65:36129)
0929 20:25:56.906295 (+ 761us) server_negotiation.cc:206] Beginning negotiation
0929 20:25:56.906304 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 20:25:56.906326 (+ 22us) server_negotiation.cc:407] Connection header received
0929 20:25:56.906408 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:25:56.906416 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:25:56.906516 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:25:56.906629 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:25:56.909316 (+ 2687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:25:56.911034 (+ 1718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:25:56.913960 (+ 2926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:25:56.914984 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:25:56.915144 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:25:56.926282 (+ 11138us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:25:56.926311 (+ 29us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:25:56.926340 (+ 29us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:25:56.926415 (+ 75us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:25:56.932074 (+ 5659us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:25:56.933439 (+ 1365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:25:56.933453 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:25:56.933459 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:25:56.933689 (+ 230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:25:56.935406 (+ 1717us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:25:56.935413 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:25:56.935419 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:25:56.936056 (+ 637us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:25:56.936470 (+ 414us) server_negotiation.cc:1035] Waiting for connection context
0929 20:25:56.938612 (+ 2142us) server_negotiation.cc:299] Negotiation successful
0929 20:25:56.938955 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":583}
I20260929 20:25:56.940352 32289 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:25:56.944435 32596 tablet_replica.cc:330] T 127e6bec99db4dba8ba6ae51ea11e36d P eedca80ed76e4368a624213c9faed862: stopping tablet replica
I20260929 20:25:56.945061 32596 raft_consensus.cc:2229] T 127e6bec99db4dba8ba6ae51ea11e36d P eedca80ed76e4368a624213c9faed862 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 20:25:56.945736 32596 raft_consensus.cc:2258] T 127e6bec99db4dba8ba6ae51ea11e36d P eedca80ed76e4368a624213c9faed862 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:25:56.947168 32598 tablet_replica.cc:330] T 127e6bec99db4dba8ba6ae51ea11e36d P 77871fa252864a14a60aefdf7b95e58d: stopping tablet replica
I20260929 20:25:56.947873 32598 raft_consensus.cc:2229] T 127e6bec99db4dba8ba6ae51ea11e36d P 77871fa252864a14a60aefdf7b95e58d [term 1 LEADER]: Raft consensus shutting down.
I20260929 20:25:56.948691 32598 raft_consensus.cc:2258] T 127e6bec99db4dba8ba6ae51ea11e36d P 77871fa252864a14a60aefdf7b95e58d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:25:56.949532 32596 ts_tablet_manager.cc:1905] T 127e6bec99db4dba8ba6ae51ea11e36d P eedca80ed76e4368a624213c9faed862: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:25:56.950881 32598 ts_tablet_manager.cc:1905] T 127e6bec99db4dba8ba6ae51ea11e36d P 77871fa252864a14a60aefdf7b95e58d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:25:56.957077 32596 ts_tablet_manager.cc:1918] T 127e6bec99db4dba8ba6ae51ea11e36d P eedca80ed76e4368a624213c9faed862: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 20:25:56.957265 32596 log.cc:1196] T 127e6bec99db4dba8ba6ae51ea11e36d P eedca80ed76e4368a624213c9faed862: Deleting WAL directory at /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713339161492-17621-0/minicluster-data/ts-1/wal/wals/127e6bec99db4dba8ba6ae51ea11e36d
I20260929 20:25:56.957819 32596 ts_tablet_manager.cc:1939] T 127e6bec99db4dba8ba6ae51ea11e36d P eedca80ed76e4368a624213c9faed862: Deleting consensus metadata
I20260929 20:25:56.958815 32598 ts_tablet_manager.cc:1918] T 127e6bec99db4dba8ba6ae51ea11e36d P 77871fa252864a14a60aefdf7b95e58d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 20:25:56.958999 32598 log.cc:1196] T 127e6bec99db4dba8ba6ae51ea11e36d P 77871fa252864a14a60aefdf7b95e58d: Deleting WAL directory at /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713339161492-17621-0/minicluster-data/ts-2/wal/wals/127e6bec99db4dba8ba6ae51ea11e36d
I20260929 20:25:56.959604 32598 ts_tablet_manager.cc:1939] T 127e6bec99db4dba8ba6ae51ea11e36d P 77871fa252864a14a60aefdf7b95e58d: Deleting consensus metadata
I20260929 20:25:56.960043 32092 catalog_manager.cc:4841] TS eedca80ed76e4368a624213c9faed862 (127.17.53.66:43527): tablet 127e6bec99db4dba8ba6ae51ea11e36d (table test-table [id=67161f25a1ca465ab65cf77a7fe59450]) successfully deleted
I20260929 20:25:56.961436 32095 catalog_manager.cc:4841] TS 77871fa252864a14a60aefdf7b95e58d (127.17.53.67:39129): tablet 127e6bec99db4dba8ba6ae51ea11e36d (table test-table [id=67161f25a1ca465ab65cf77a7fe59450]) successfully deleted
I20260929 20:25:57.047981 32597 tablet_replica.cc:330] T 127e6bec99db4dba8ba6ae51ea11e36d P ad1b6c08928b446ca0b8539b56223f57: stopping tablet replica
I20260929 20:25:57.048451 32597 raft_consensus.cc:2229] T 127e6bec99db4dba8ba6ae51ea11e36d P ad1b6c08928b446ca0b8539b56223f57 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 20:25:57.048980 17621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu with pid 32145
I20260929 20:25:57.049007 32597 raft_consensus.cc:2258] T 127e6bec99db4dba8ba6ae51ea11e36d P ad1b6c08928b446ca0b8539b56223f57 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260929 20:25:57.068945 32095 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.17.53.65:36129 (error 108)
I20260929 20:25:57.069278 17621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu pid 32298
I20260929 20:25:57.073467 32571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:25:57.071068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.65:36129 (local address 127.0.0.1:39640)
0929 20:25:57.073159 (+ 2091us) negotiation.cc:107] Waiting for socket to connect
0929 20:25:57.073296 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.53.65:36129: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1964}
W20260929 20:25:57.073722 32095 catalog_manager.cc:4584] TS ad1b6c08928b446ca0b8539b56223f57 (127.17.53.65:36129): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 127e6bec99db4dba8ba6ae51ea11e36d: Network error: Client connection negotiation failed: client connection to 127.17.53.65:36129: connect: Connection refused (error 111)
I20260929 20:25:57.088762 32570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:25:57.070134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58324 (local address 127.17.53.66:43527)
0929 20:25:57.070333 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0929 20:25:57.070339 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:25:57.070726 (+ 387us) server_negotiation.cc:407] Connection header received
0929 20:25:57.070848 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:25:57.070876 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:25:57.070958 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:25:57.071072 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:25:57.073039 (+ 1967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:25:57.074243 (+ 1204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:25:57.076433 (+ 2190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:25:57.077087 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:25:57.077217 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:25:57.080751 (+ 3534us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:25:57.080771 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:25:57.080789 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:25:57.080827 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:25:57.084787 (+ 3960us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:25:57.085655 (+ 868us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:25:57.085664 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:25:57.085668 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:25:57.085816 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:25:57.086267 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:25:57.086271 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:25:57.086275 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:25:57.086816 (+ 541us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:25:57.087135 (+ 319us) server_negotiation.cc:1035] Waiting for connection context
0929 20:25:57.088146 (+ 1011us) server_negotiation.cc:299] Negotiation successful
0929 20:25:57.088383 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260929 20:25:57.089558 32421 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:25:57.127656 32571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:25:57.127062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.65:36129 (local address 127.0.0.1:39642)
0929 20:25:57.127347 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0929 20:25:57.127467 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.53.65:36129: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159}
I20260929 20:25:57.128800 32571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:25:57.128455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.65:36129 (local address 127.0.0.1:39658)
0929 20:25:57.128580 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0929 20:25:57.128667 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.53.65:36129: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260929 20:25:57.189450 32571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:25:57.188956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.65:36129 (local address 127.0.0.1:39674)
0929 20:25:57.189164 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0929 20:25:57.189268 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.53.65:36129: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260929 20:25:57.190701 32571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:25:57.190312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.65:36129 (local address 127.0.0.1:39684)
0929 20:25:57.190477 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0929 20:25:57.190569 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.53.65:36129: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260929 20:25:57.195005 17621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu with pid 32298
I20260929 20:25:57.214048 17621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu pid 32429
I20260929 20:25:57.230757 32572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:25:57.214946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54654 (local address 127.17.53.67:39129)
0929 20:25:57.215146 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0929 20:25:57.215151 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:25:57.215261 (+ 110us) server_negotiation.cc:407] Connection header received
0929 20:25:57.215446 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:25:57.215450 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:25:57.215533 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:25:57.215611 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:25:57.217619 (+ 2008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:25:57.218408 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:25:57.220464 (+ 2056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:25:57.220896 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:25:57.220988 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:25:57.224589 (+ 3601us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:25:57.224601 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:25:57.224616 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:25:57.224648 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:25:57.227786 (+ 3138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:25:57.228658 (+ 872us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:25:57.228665 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:25:57.228667 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:25:57.228823 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:25:57.229241 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:25:57.229244 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:25:57.229246 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:25:57.229585 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:25:57.229826 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0929 20:25:57.230233 (+ 407us) server_negotiation.cc:299] Negotiation successful
0929 20:25:57.230397 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260929 20:25:57.231468 32552 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:25:57.271385 32571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:25:57.270767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.65:36129 (local address 127.0.0.1:39690)
0929 20:25:57.271066 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0929 20:25:57.271186 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.53.65:36129: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260929 20:25:57.272709 32571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:25:57.272282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.65:36129 (local address 127.0.0.1:39702)
0929 20:25:57.272503 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0929 20:25:57.272578 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.53.65:36129: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101}
I20260929 20:25:57.357182 17621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu with pid 32429
I20260929 20:25:57.377269 17621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu pid 32075
I20260929 20:25:57.378387 32138 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:25:57.492229 32571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:25:57.491125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.65:36129 (local address 127.0.0.1:39718)
0929 20:25:57.491709 (+ 584us) negotiation.cc:107] Waiting for socket to connect
0929 20:25:57.491820 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.53.65:36129: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":392}
I20260929 20:25:57.493148 17621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu with pid 32075
I20260929 20:25:57.517184 17621 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
/tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713339161492-17621-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713339161492-17621-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713339161492-17621-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713339161492-17621-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.53.126:37213
--webserver_interface=127.17.53.126
--webserver_port=40845
--keytab_file=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.126.keytab
--principal=kudu/127.17.53.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.53.84:41567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.53.126:37213
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713339161492-17621-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713339161492-17621-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713339161492-17621-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:25:57.846371 32604 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:25:57.846665 32604 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:25:57.846822 32604 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:25:57.853992 32604 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:25:57.854094 32604 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:25:57.854202 32604 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:25:57.854296 32604 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:25:57.854359 32604 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:25:57.864291 32604 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.53.84:41567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713339161492-17621-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713339161492-17621-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.53.126:37213
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.126.keytab
--principal=kudu/127.17.53.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.53.126:37213
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713339161492-17621-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.53.126
--webserver_port=40845
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713339161492-17621-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:25:57.865579 32604 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:25:57.866899 32604 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:25:57.878847 32609 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:25:57.878893 32610 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:25:57.878950 32612 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:25:57.880044 32604 server_base.cc:1034] running on GCE node
I20260929 20:25:57.880872 32604 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:25:57.882483 32604 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:25:57.883733 32604 hybrid_clock.cc:648] HybridClock initialized: now 1790713557883671 us; error 55 us; skew 500 ppm
I20260929 20:25:57.884109 32604 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:25:57 dist-test-slave-zbnc krb5kdc[31522](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713557, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.53.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:25:57.887163 32604 init.cc:377] Logged in from keytab as kudu/127.17.53.126@KRBTEST.COM (short username kudu)
I20260929 20:25:57.888698 32604 webserver.cc:460] Webserver started at http://127.17.53.126:40845/ using document root <none> and password file <none>
I20260929 20:25:57.889400 32604 fs_manager.cc:359] Metadata directory not provided
I20260929 20:25:57.889550 32604 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:25:57.894933 32604 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260929 20:25:57.897698 32619 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:25:57.899080 32604 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260929 20:25:57.899299 32604 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713339161492-17621-0/minicluster-data/master-0/data,/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713339161492-17621-0/minicluster-data/master-0/wal
uuid: "7caaae55b00c4da99832e09118f54e5e"
format_stamp: "Formatted at 2026-09-29 20:25:52 on dist-test-slave-zbnc"
server_key: "556bb7331b6e6cd015f30def5b023699"
server_key_iv: "e08504e502dda6de1daf7acf50e383a6"
server_key_version: "encryptionkey@0"
I20260929 20:25:57.900296 32604 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713339161492-17621-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713339161492-17621-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713339161492-17621-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:25:57.946048 32604 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:25:57.950536 32604 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:25:57.950832 32604 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:25:57.970316 32604 rpc_server.cc:307] RPC server started. Bound to: 127.17.53.126:37213
I20260929 20:25:57.970336 32670 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.53.126:37213 every 8 connection(s)
I20260929 20:25:57.971928 32604 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713339161492-17621-0/minicluster-data/master-0/data/info.pb
I20260929 20:25:57.978883 32671 sys_catalog.cc:263] Verifying existing consensus state
I20260929 20:25:57.979805 17621 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu as pid 32604
I20260929 20:25:57.980832 17621 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
/tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713339161492-17621-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713339161492-17621-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713339161492-17621-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713339161492-17621-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.53.65:36129
--local_ip_for_outbound_sockets=127.17.53.65
--tserver_master_addrs=127.17.53.126:37213
--webserver_port=46229
--webserver_interface=127.17.53.65
--keytab_file=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.65.keytab
--principal=kudu/127.17.53.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.53.84:41567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713339161492-17621-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713339161492-17621-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713339161492-17621-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 20:25:57.983991 32671 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7caaae55b00c4da99832e09118f54e5e: Bootstrap starting.
I20260929 20:25:58.006263 32671 log.cc:824] T 00000000000000000000000000000000 P 7caaae55b00c4da99832e09118f54e5e: Log is configured to *not* fsync() on all Append() calls
I20260929 20:25:58.022603 32671 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7caaae55b00c4da99832e09118f54e5e: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 20:25:58.023252 32671 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7caaae55b00c4da99832e09118f54e5e: Bootstrap complete.
I20260929 20:25:58.032614 32671 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7caaae55b00c4da99832e09118f54e5e [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7caaae55b00c4da99832e09118f54e5e" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 37213 } }
I20260929 20:25:58.033659 32671 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7caaae55b00c4da99832e09118f54e5e [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7caaae55b00c4da99832e09118f54e5e, State: Initialized, Role: FOLLOWER
I20260929 20:25:58.034351 32671 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7caaae55b00c4da99832e09118f54e5e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7caaae55b00c4da99832e09118f54e5e" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 37213 } }
I20260929 20:25:58.034600 32671 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7caaae55b00c4da99832e09118f54e5e [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 20:25:58.034725 32671 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7caaae55b00c4da99832e09118f54e5e [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 20:25:58.034942 32671 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7caaae55b00c4da99832e09118f54e5e [term 2 FOLLOWER]: Advancing to term 3
I20260929 20:25:58.038738 32671 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7caaae55b00c4da99832e09118f54e5e [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7caaae55b00c4da99832e09118f54e5e" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 37213 } }
I20260929 20:25:58.039336 32671 leader_election.cc:302] T 00000000000000000000000000000000 P 7caaae55b00c4da99832e09118f54e5e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7caaae55b00c4da99832e09118f54e5e; no voters:
I20260929 20:25:58.040005 32671 leader_election.cc:288] T 00000000000000000000000000000000 P 7caaae55b00c4da99832e09118f54e5e [CANDIDATE]: Term 3 election: Requested vote from peers
I20260929 20:25:58.040164 32675 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7caaae55b00c4da99832e09118f54e5e [term 3 FOLLOWER]: Leader election won for term 3
I20260929 20:25:58.041326 32675 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7caaae55b00c4da99832e09118f54e5e [term 3 LEADER]: Becoming Leader. State: Replica: 7caaae55b00c4da99832e09118f54e5e, State: Running, Role: LEADER
I20260929 20:25:58.041882 32675 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7caaae55b00c4da99832e09118f54e5e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7caaae55b00c4da99832e09118f54e5e" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 37213 } }
I20260929 20:25:58.042243 32671 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7caaae55b00c4da99832e09118f54e5e [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:25:58.044740 32677 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7caaae55b00c4da99832e09118f54e5e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7caaae55b00c4da99832e09118f54e5e. Latest consensus state: current_term: 3 leader_uuid: "7caaae55b00c4da99832e09118f54e5e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7caaae55b00c4da99832e09118f54e5e" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 37213 } } }
I20260929 20:25:58.045162 32677 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7caaae55b00c4da99832e09118f54e5e [sys.catalog]: This master's current role is: LEADER
I20260929 20:25:58.045676 32676 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7caaae55b00c4da99832e09118f54e5e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "7caaae55b00c4da99832e09118f54e5e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7caaae55b00c4da99832e09118f54e5e" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 37213 } } }
I20260929 20:25:58.046013 32676 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7caaae55b00c4da99832e09118f54e5e [sys.catalog]: This master's current role is: LEADER
I20260929 20:25:58.051627 32680 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:25:58.060333 32680 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:25:58.061832 32680 catalog_manager.cc:1259] Loaded cluster ID: 4893bb9645e249e4b64ef32dfa3ad00f
I20260929 20:25:58.061929 32680 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260929 20:25:58.063956 32680 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f3567a8d9ad google::LogMessage::Fail() at ??:0
@ 0x7f3567a9088a google::LogMessage::SendToLog() at ??:0
@ 0x7f3567a8d4de google::LogMessage::Flush() at ??:0
@ 0x7f3567a8ea19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f357d4c4512 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f357d4c1976 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f357d549610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
I20260929 20:25:58.103433 32694 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
@ 0x7f357d549439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f356a6d9345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f356a6dc91c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f356a6dc799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f356a6b7eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f356d9916db start_thread at ??:0
@ 0x7f356692761f clone at ??:0
W20260929 20:25:58.367749 32673 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:25:58.368070 32673 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:25:58.368244 32673 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:25:58.376243 32673 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:25:58.376390 32673 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:25:58.376567 32673 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.53.65
I20260929 20:25:58.387439 32673 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.53.84:41567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713339161492-17621-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713339161492-17621-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.65.keytab
--principal=kudu/127.17.53.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.53.65:36129
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713339161492-17621-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.17.53.65
--webserver_port=46229
--tserver_master_addrs=127.17.53.126:37213
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.53.65
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713339161492-17621-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:25:58.388842 32673 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:25:58.390232 32673 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:25:58.404275 32712 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:25:58.404315 32715 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:25:58.404290 32713 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:25:58.405061 32673 server_base.cc:1034] running on GCE node
I20260929 20:25:58.405709 32673 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:25:58.406618 32673 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:25:58.407831 32673 hybrid_clock.cc:648] HybridClock initialized: now 1790713558407776 us; error 53 us; skew 500 ppm
I20260929 20:25:58.408123 32673 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:25:58 dist-test-slave-zbnc krb5kdc[31522](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713558, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.53.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:25:58.411339 32673 init.cc:377] Logged in from keytab as kudu/127.17.53.65@KRBTEST.COM (short username kudu)
I20260929 20:25:58.412922 32673 webserver.cc:460] Webserver started at http://127.17.53.65:46229/ using document root <none> and password file <none>
I20260929 20:25:58.413643 32673 fs_manager.cc:359] Metadata directory not provided
I20260929 20:25:58.413782 32673 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:25:58.419371 32673 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260929 20:25:58.424471 32722 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:25:58.426029 32673 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 20:25:58.426230 32673 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713339161492-17621-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713339161492-17621-0/minicluster-data/ts-0/wal
uuid: "ad1b6c08928b446ca0b8539b56223f57"
format_stamp: "Formatted at 2026-09-29 20:25:52 on dist-test-slave-zbnc"
server_key: "ea98d3192c7fc95189919bf5c74d03ae"
server_key_iv: "03dc1c8d9834e63fb29a049126bdefe3"
server_key_version: "encryptionkey@0"
I20260929 20:25:58.427358 32673 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713339161492-17621-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713339161492-17621-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713339161492-17621-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:25:58.456874 32673 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:25:58.461084 32673 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:25:58.461370 32673 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:25:58.462409 32673 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:25:58.464267 32729 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260929 20:25:58.470031 32673 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260929 20:25:58.470139 32673 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.001s sys 0.000s
I20260929 20:25:58.470324 32673 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260929 20:25:58.473562 32673 ts_tablet_manager.cc:610] Registered 1 tablets
I20260929 20:25:58.473635 32673 ts_tablet_manager.cc:589] Time spent register tablets: real 0.003s user 0.001s sys 0.000s
I20260929 20:25:58.474088 32729 tablet_bootstrap.cc:492] T 127e6bec99db4dba8ba6ae51ea11e36d P ad1b6c08928b446ca0b8539b56223f57: Bootstrap starting.
I20260929 20:25:58.502020 32729 log.cc:824] T 127e6bec99db4dba8ba6ae51ea11e36d P ad1b6c08928b446ca0b8539b56223f57: Log is configured to *not* fsync() on all Append() calls
I20260929 20:25:58.514977 32729 tablet_bootstrap.cc:492] T 127e6bec99db4dba8ba6ae51ea11e36d P ad1b6c08928b446ca0b8539b56223f57: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 20:25:58.515441 32673 rpc_server.cc:307] RPC server started. Bound to: 127.17.53.65:36129
I20260929 20:25:58.515467 368 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.53.65:36129 every 8 connection(s)
I20260929 20:25:58.515623 32729 tablet_bootstrap.cc:492] T 127e6bec99db4dba8ba6ae51ea11e36d P ad1b6c08928b446ca0b8539b56223f57: Bootstrap complete.
I20260929 20:25:58.516682 32673 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713339161492-17621-0/minicluster-data/ts-0/data/info.pb
I20260929 20:25:58.516968 32729 ts_tablet_manager.cc:1397] T 127e6bec99db4dba8ba6ae51ea11e36d P ad1b6c08928b446ca0b8539b56223f57: Time spent bootstrapping tablet: real 0.043s user 0.021s sys 0.019s
I20260929 20:25:58.518482 17621 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu as pid 32673
I20260929 20:25:58.519326 17621 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
/tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713339161492-17621-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713339161492-17621-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713339161492-17621-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713339161492-17621-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.53.66:43527
--local_ip_for_outbound_sockets=127.17.53.66
--tserver_master_addrs=127.17.53.126:37213
--webserver_port=45857
--webserver_interface=127.17.53.66
--keytab_file=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.66.keytab
--principal=kudu/127.17.53.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.53.84:41567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713339161492-17621-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713339161492-17621-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713339161492-17621-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 20:25:58.522745 371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:25:58.520189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:37213 (local address 127.17.53.65:55021)
0929 20:25:58.521175 (+ 986us) negotiation.cc:107] Waiting for socket to connect
0929 20:25:58.521526 (+ 351us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.53.126:37213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":654,"thread_start_us":502,"threads_started":1}
W20260929 20:25:58.523509 369 heartbeater.cc:644] Failed to heartbeat to 127.17.53.126:37213 (0 consecutive failures): Network error: Failed to ping master at 127.17.53.126:37213: Client connection negotiation failed: client connection to 127.17.53.126:37213: connect: Connection refused (error 111)
I20260929 20:25:58.524561 371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:25:58.524096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:37213 (local address 127.17.53.65:60477)
0929 20:25:58.524312 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0929 20:25:58.524412 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.53.126:37213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260929 20:25:58.525812 371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:25:58.525332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:37213 (local address 127.17.53.65:59811)
0929 20:25:58.525513 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0929 20:25:58.525629 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.53.126:37213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260929 20:25:58.526127 369 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260929 20:25:58.526265 32729 raft_consensus.cc:348] T 127e6bec99db4dba8ba6ae51ea11e36d P ad1b6c08928b446ca0b8539b56223f57 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad1b6c08928b446ca0b8539b56223f57" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 36129 } } peers { permanent_uuid: "eedca80ed76e4368a624213c9faed862" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 43527 } } peers { permanent_uuid: "77871fa252864a14a60aefdf7b95e58d" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 39129 } }
I20260929 20:25:58.527225 32729 raft_consensus.cc:729] T 127e6bec99db4dba8ba6ae51ea11e36d P ad1b6c08928b446ca0b8539b56223f57 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad1b6c08928b446ca0b8539b56223f57, State: Initialized, Role: FOLLOWER
I20260929 20:25:58.527788 32729 consensus_queue.cc:260] T 127e6bec99db4dba8ba6ae51ea11e36d P ad1b6c08928b446ca0b8539b56223f57 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad1b6c08928b446ca0b8539b56223f57" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 36129 } } peers { permanent_uuid: "eedca80ed76e4368a624213c9faed862" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 43527 } } peers { permanent_uuid: "77871fa252864a14a60aefdf7b95e58d" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 39129 } }
I20260929 20:25:58.529394 32729 ts_tablet_manager.cc:1428] T 127e6bec99db4dba8ba6ae51ea11e36d P ad1b6c08928b446ca0b8539b56223f57: Time spent starting tablet: real 0.012s user 0.013s sys 0.001s
I20260929 20:25:58.529485 371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:25:58.529052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:37213 (local address 127.17.53.65:52463)
0929 20:25:58.529220 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0929 20:25:58.529307 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.53.126:37213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260929 20:25:58.530400 371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:25:58.530013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:37213 (local address 127.17.53.65:58423)
0929 20:25:58.530170 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0929 20:25:58.530251 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.53.126:37213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260929 20:25:58.846912 372 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:25:58.847200 372 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:25:58.847375 372 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:25:58.854648 372 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:25:58.854754 372 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:25:58.854935 372 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.53.66
I20260929 20:25:58.864540 372 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.53.84:41567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713339161492-17621-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713339161492-17621-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.66.keytab
--principal=kudu/127.17.53.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.53.66:43527
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713339161492-17621-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.17.53.66
--webserver_port=45857
--tserver_master_addrs=127.17.53.126:37213
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.53.66
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713339161492-17621-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:25:58.865880 372 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:25:58.867247 372 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:25:58.880103 381 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:25:58.880213 379 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:25:58.880199 378 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:25:58.881101 372 server_base.cc:1034] running on GCE node
I20260929 20:25:58.881716 372 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:25:58.882726 372 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:25:58.883948 372 hybrid_clock.cc:648] HybridClock initialized: now 1790713558883905 us; error 44 us; skew 500 ppm
I20260929 20:25:58.884274 372 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:25:58 dist-test-slave-zbnc krb5kdc[31522](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713558, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.53.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:25:58.887328 372 init.cc:377] Logged in from keytab as kudu/127.17.53.66@KRBTEST.COM (short username kudu)
I20260929 20:25:58.888762 372 webserver.cc:460] Webserver started at http://127.17.53.66:45857/ using document root <none> and password file <none>
I20260929 20:25:58.889461 372 fs_manager.cc:359] Metadata directory not provided
I20260929 20:25:58.889592 372 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:25:58.895051 372 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260929 20:25:58.897740 388 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:25:58.899168 372 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 20:25:58.899341 372 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713339161492-17621-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713339161492-17621-0/minicluster-data/ts-1/wal
uuid: "eedca80ed76e4368a624213c9faed862"
format_stamp: "Formatted at 2026-09-29 20:25:53 on dist-test-slave-zbnc"
server_key: "7b18e8c3cf0d8876f9869e66f8cc41a4"
server_key_iv: "5a1d701cc0dad48f1c12dd09c3debb72"
server_key_version: "encryptionkey@0"
I20260929 20:25:58.900326 372 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713339161492-17621-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713339161492-17621-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713339161492-17621-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:25:58.930182 372 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:25:58.934698 372 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:25:58.935014 372 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:25:58.936036 372 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:25:58.937387 372 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:25:58.937498 372 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:25:58.937628 372 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:25:58.937728 372 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:25:58.971415 372 rpc_server.cc:307] RPC server started. Bound to: 127.17.53.66:43527
I20260929 20:25:58.971508 501 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.53.66:43527 every 8 connection(s)
I20260929 20:25:58.972988 372 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713339161492-17621-0/minicluster-data/ts-1/data/info.pb
I20260929 20:25:58.974933 17621 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu as pid 372
I20260929 20:25:58.975718 17621 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
/tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713339161492-17621-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713339161492-17621-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713339161492-17621-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713339161492-17621-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.53.67:39129
--local_ip_for_outbound_sockets=127.17.53.67
--tserver_master_addrs=127.17.53.126:37213
--webserver_port=40081
--webserver_interface=127.17.53.67
--keytab_file=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.67.keytab
--principal=kudu/127.17.53.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.53.84:41567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713339161492-17621-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713339161492-17621-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713339161492-17621-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 20:25:58.978294 504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:25:58.976430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:37213 (local address 127.17.53.66:47965)
0929 20:25:58.977196 (+ 766us) negotiation.cc:107] Waiting for socket to connect
0929 20:25:58.977428 (+ 232us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.53.126:37213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":505,"thread_start_us":301,"threads_started":1}
W20260929 20:25:58.979105 502 heartbeater.cc:644] Failed to heartbeat to 127.17.53.126:37213 (0 consecutive failures): Network error: Failed to ping master at 127.17.53.126:37213: Client connection negotiation failed: client connection to 127.17.53.126:37213: connect: Connection refused (error 111)
I20260929 20:25:58.980016 504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:25:58.979677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:37213 (local address 127.17.53.66:41331)
0929 20:25:58.979834 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0929 20:25:58.979916 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.53.126:37213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260929 20:25:58.980944 504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:25:58.980647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:37213 (local address 127.17.53.66:42941)
0929 20:25:58.980770 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0929 20:25:58.980840 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.53.126:37213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260929 20:25:58.982268 502 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 20:25:59.314567 505 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:25:59.314991 505 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:25:59.315243 505 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:25:59.323550 505 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:25:59.323714 505 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:25:59.323913 505 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.53.67
I20260929 20:25:59.335050 505 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.53.84:41567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713339161492-17621-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713339161492-17621-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.67.keytab
--principal=kudu/127.17.53.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.53.67:39129
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713339161492-17621-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.17.53.67
--webserver_port=40081
--tserver_master_addrs=127.17.53.126:37213
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.53.67
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713339161492-17621-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:25:59.336598 505 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:25:59.338029 505 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:25:59.352222 511 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:25:59.352190 510 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:25:59.352363 513 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:25:59.352953 505 server_base.cc:1034] running on GCE node
I20260929 20:25:59.353673 505 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:25:59.354595 505 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:25:59.355803 505 hybrid_clock.cc:648] HybridClock initialized: now 1790713559355723 us; error 71 us; skew 500 ppm
I20260929 20:25:59.356091 505 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:25:59 dist-test-slave-zbnc krb5kdc[31522](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713559, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.53.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:25:59.359261 505 init.cc:377] Logged in from keytab as kudu/127.17.53.67@KRBTEST.COM (short username kudu)
I20260929 20:25:59.360647 505 webserver.cc:460] Webserver started at http://127.17.53.67:40081/ using document root <none> and password file <none>
I20260929 20:25:59.361351 505 fs_manager.cc:359] Metadata directory not provided
I20260929 20:25:59.361454 505 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:25:59.366848 505 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260929 20:25:59.369752 520 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:25:59.371292 505 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 20:25:59.371428 505 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713339161492-17621-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713339161492-17621-0/minicluster-data/ts-2/wal
uuid: "77871fa252864a14a60aefdf7b95e58d"
format_stamp: "Formatted at 2026-09-29 20:25:53 on dist-test-slave-zbnc"
server_key: "283af3aeaffef06b486bfbab35cd9520"
server_key_iv: "92e1a9b45c86a47eae578b5751b56ce6"
server_key_version: "encryptionkey@0"
I20260929 20:25:59.372387 505 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713339161492-17621-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713339161492-17621-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713339161492-17621-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:25:59.397336 505 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:25:59.401906 505 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:25:59.402135 505 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:25:59.403201 505 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:25:59.404635 505 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:25:59.404788 505 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:25:59.404875 505 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:25:59.404930 505 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:25:59.438678 505 rpc_server.cc:307] RPC server started. Bound to: 127.17.53.67:39129
I20260929 20:25:59.438787 632 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.53.67:39129 every 8 connection(s)
I20260929 20:25:59.440229 505 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790713339161492-17621-0/minicluster-data/ts-2/data/info.pb
I20260929 20:25:59.445137 635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:25:59.443325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:37213 (local address 127.17.53.67:56329)
0929 20:25:59.444037 (+ 712us) negotiation.cc:107] Waiting for socket to connect
0929 20:25:59.444273 (+ 236us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.53.126:37213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":278,"threads_started":1}
W20260929 20:25:59.445866 633 heartbeater.cc:644] Failed to heartbeat to 127.17.53.126:37213 (0 consecutive failures): Network error: Failed to ping master at 127.17.53.126:37213: Client connection negotiation failed: client connection to 127.17.53.126:37213: connect: Connection refused (error 111)
I20260929 20:25:59.446645 635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:25:59.446333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:37213 (local address 127.17.53.67:33631)
0929 20:25:59.446486 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0929 20:25:59.446548 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.53.126:37213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260929 20:25:59.447527 635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:25:59.447200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:37213 (local address 127.17.53.67:45117)
0929 20:25:59.447349 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0929 20:25:59.447431 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.53.126:37213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260929 20:25:59.447780 633 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260929 20:25:59.450315 17621 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu as pid 505
I20260929 20:25:59.531997 637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:25:59.531194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:37213 (local address 127.17.53.65:42949)
0929 20:25:59.531642 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0929 20:25:59.531800 (+ 158us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.53.126:37213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":211,"threads_started":1}
I20260929 20:25:59.982076 638 raft_consensus.cc:482] T 127e6bec99db4dba8ba6ae51ea11e36d P ad1b6c08928b446ca0b8539b56223f57 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 20:25:59.982340 638 raft_consensus.cc:504] T 127e6bec99db4dba8ba6ae51ea11e36d P ad1b6c08928b446ca0b8539b56223f57 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad1b6c08928b446ca0b8539b56223f57" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 36129 } } peers { permanent_uuid: "eedca80ed76e4368a624213c9faed862" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 43527 } } peers { permanent_uuid: "77871fa252864a14a60aefdf7b95e58d" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 39129 } }
I20260929 20:25:59.983695 639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:25:59.982823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:37213 (local address 127.17.53.66:56375)
0929 20:25:59.983378 (+ 555us) negotiation.cc:107] Waiting for socket to connect
0929 20:25:59.983479 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.53.126:37213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":431,"thread_start_us":263,"threads_started":1}
I20260929 20:25:59.984148 638 leader_election.cc:288] T 127e6bec99db4dba8ba6ae51ea11e36d P ad1b6c08928b446ca0b8539b56223f57 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers eedca80ed76e4368a624213c9faed862 (127.17.53.66:43527), 77871fa252864a14a60aefdf7b95e58d (127.17.53.67:39129)
Sep 29 20:25:59 dist-test-slave-zbnc krb5kdc[31522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713558, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.53.65@KRBTEST.COM for kudu/127.17.53.66@KRBTEST.COM
Sep 29 20:25:59 dist-test-slave-zbnc krb5kdc[31522](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713558, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.53.65@KRBTEST.COM for kudu/127.17.53.67@KRBTEST.COM
I20260929 20:26:00.007609 641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:25:59.984726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.53.65:50999 (local address 127.17.53.66:43527)
0929 20:25:59.985986 (+ 1260us) server_negotiation.cc:206] Beginning negotiation
0929 20:25:59.986004 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0929 20:25:59.986054 (+ 50us) server_negotiation.cc:407] Connection header received
0929 20:25:59.988255 (+ 2201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:25:59.988268 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:25:59.988747 (+ 479us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:25:59.989500 (+ 753us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:25:59.991300 (+ 1800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:25:59.993096 (+ 1796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:25:59.994812 (+ 1716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:25:59.995491 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:25:59.995705 (+ 214us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:25:59.999970 (+ 4265us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:25:59.999979 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:25:59.999996 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:26:00.000033 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:26:00.003623 (+ 3590us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:26:00.004601 (+ 978us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:26:00.004617 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:26:00.004623 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:26:00.004779 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:26:00.005223 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:26:00.005229 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:26:00.005232 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:26:00.005832 (+ 600us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:26:00.006025 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0929 20:26:00.006813 (+ 788us) server_negotiation.cc:299] Negotiation successful
0929 20:26:00.007190 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1091,"thread_start_us":964,"threads_started":1}
I20260929 20:26:00.007661 642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:25:59.984800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.53.65:35703 (local address 127.17.53.67:39129)
0929 20:25:59.986226 (+ 1426us) server_negotiation.cc:206] Beginning negotiation
0929 20:25:59.986238 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 20:25:59.986644 (+ 406us) server_negotiation.cc:407] Connection header received
0929 20:25:59.988713 (+ 2069us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:25:59.988723 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:25:59.989133 (+ 410us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:25:59.989457 (+ 324us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:25:59.991243 (+ 1786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:25:59.993096 (+ 1853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:25:59.994805 (+ 1709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:25:59.995503 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:25:59.995707 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:26:00.000125 (+ 4418us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:26:00.000132 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:26:00.000152 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:26:00.000232 (+ 80us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:26:00.003530 (+ 3298us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:26:00.004616 (+ 1086us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:26:00.004623 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:26:00.004629 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:26:00.004796 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:26:00.005314 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:26:00.005319 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:26:00.005324 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:26:00.005824 (+ 500us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:26:00.006017 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0929 20:26:00.006799 (+ 782us) server_negotiation.cc:299] Negotiation successful
0929 20:26:00.007235 (+ 436us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1264,"thread_start_us":1142,"threads_started":1}
I20260929 20:26:00.007792 637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:25:59.984341 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.66:43527 (local address 127.17.53.65:50999)
0929 20:25:59.984523 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0929 20:25:59.984565 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 20:25:59.985149 (+ 584us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:25:59.989274 (+ 4125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:25:59.989282 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:25:59.990310 (+ 1028us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:25:59.991052 (+ 742us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:25:59.991073 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:25:59.993280 (+ 2207us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:25:59.993283 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:25:59.994662 (+ 1379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:25:59.994674 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:25:59.995750 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:25:59.995756 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:25:59.996009 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:25:59.997046 (+ 1037us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:25:59.997073 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:25:59.999597 (+ 2524us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:26:00.003788 (+ 4191us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:26:00.003793 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:26:00.003797 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:26:00.004460 (+ 663us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:26:00.004918 (+ 458us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:26:00.004923 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:26:00.004926 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:26:00.005109 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:26:00.005974 (+ 865us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:26:00.005982 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:26:00.006307 (+ 325us) client_negotiation.cc:769] Sending connection context
0929 20:26:00.006589 (+ 282us) client_negotiation.cc:240] Negotiation successful
0929 20:26:00.007406 (+ 817us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62,"mutex_wait_us":479,"spinlock_wait_cycles":14080}
I20260929 20:26:00.008000 640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:25:59.984533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.67:39129 (local address 127.17.53.65:35703)
0929 20:25:59.986537 (+ 2004us) negotiation.cc:107] Waiting for socket to connect
0929 20:25:59.986551 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 20:25:59.986646 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:25:59.989418 (+ 2772us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:25:59.989422 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:25:59.990238 (+ 816us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:25:59.991051 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:25:59.991072 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:25:59.993326 (+ 2254us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:25:59.993330 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:25:59.994656 (+ 1326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:25:59.994667 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:25:59.995750 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:25:59.995758 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:25:59.996012 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:25:59.997131 (+ 1119us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:25:59.997151 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:25:59.999865 (+ 2714us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:26:00.003706 (+ 3841us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:26:00.003720 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:26:00.003732 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:26:00.004468 (+ 736us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:26:00.005014 (+ 546us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:26:00.005019 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:26:00.005024 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:26:00.005189 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:26:00.005971 (+ 782us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:26:00.005979 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:26:00.006302 (+ 323us) client_negotiation.cc:769] Sending connection context
0929 20:26:00.006586 (+ 284us) client_negotiation.cc:240] Negotiation successful
0929 20:26:00.007230 (+ 644us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1896,"mutex_wait_us":129,"spinlock_wait_cycles":25472,"thread_start_us":1758,"threads_started":1}
W20260929 20:26:00.011339 32723 leader_election.cc:341] T 127e6bec99db4dba8ba6ae51ea11e36d P ad1b6c08928b446ca0b8539b56223f57 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer eedca80ed76e4368a624213c9faed862 (127.17.53.66:43527): Not found: Tablet not found: 127e6bec99db4dba8ba6ae51ea11e36d
W20260929 20:26:00.011756 32726 leader_election.cc:341] T 127e6bec99db4dba8ba6ae51ea11e36d P ad1b6c08928b446ca0b8539b56223f57 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 77871fa252864a14a60aefdf7b95e58d (127.17.53.67:39129): Not found: Tablet not found: 127e6bec99db4dba8ba6ae51ea11e36d
I20260929 20:26:00.011905 32726 leader_election.cc:302] T 127e6bec99db4dba8ba6ae51ea11e36d P ad1b6c08928b446ca0b8539b56223f57 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ad1b6c08928b446ca0b8539b56223f57; no voters: 77871fa252864a14a60aefdf7b95e58d, eedca80ed76e4368a624213c9faed862
I20260929 20:26:00.012367 638 raft_consensus.cc:2726] T 127e6bec99db4dba8ba6ae51ea11e36d P ad1b6c08928b446ca0b8539b56223f57 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 20:26:00.449321 643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:26:00.448342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:37213 (local address 127.17.53.67:48657)
0929 20:26:00.448917 (+ 575us) negotiation.cc:107] Waiting for socket to connect
0929 20:26:00.449061 (+ 144us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.53.126:37213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":263,"threads_started":1}
I20260929 20:26:00.533479 644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:26:00.532801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:37213 (local address 127.17.53.65:38331)
0929 20:26:00.533234 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0929 20:26:00.533316 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.53.126:37213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":321,"thread_start_us":218,"threads_started":1}
I20260929 20:26:00.985270 645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:26:00.984490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:37213 (local address 127.17.53.66:47019)
0929 20:26:00.985033 (+ 543us) negotiation.cc:107] Waiting for socket to connect
0929 20:26:00.985113 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.53.126:37213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":343,"threads_started":1}
I20260929 20:26:01.451036 646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:26:01.450274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:37213 (local address 127.17.53.67:38025)
0929 20:26:01.450722 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0929 20:26:01.450803 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.53.126:37213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":245,"threads_started":1}
I20260929 20:26:01.525854 647 raft_consensus.cc:482] T 127e6bec99db4dba8ba6ae51ea11e36d P ad1b6c08928b446ca0b8539b56223f57 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 20:26:01.526074 647 raft_consensus.cc:504] T 127e6bec99db4dba8ba6ae51ea11e36d P ad1b6c08928b446ca0b8539b56223f57 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad1b6c08928b446ca0b8539b56223f57" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 36129 } } peers { permanent_uuid: "eedca80ed76e4368a624213c9faed862" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 43527 } } peers { permanent_uuid: "77871fa252864a14a60aefdf7b95e58d" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 39129 } }
I20260929 20:26:01.526671 647 leader_election.cc:288] T 127e6bec99db4dba8ba6ae51ea11e36d P ad1b6c08928b446ca0b8539b56223f57 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers eedca80ed76e4368a624213c9faed862 (127.17.53.66:43527), 77871fa252864a14a60aefdf7b95e58d (127.17.53.67:39129)
W20260929 20:26:01.527778 32726 leader_election.cc:341] T 127e6bec99db4dba8ba6ae51ea11e36d P ad1b6c08928b446ca0b8539b56223f57 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 77871fa252864a14a60aefdf7b95e58d (127.17.53.67:39129): Not found: Tablet not found: 127e6bec99db4dba8ba6ae51ea11e36d
W20260929 20:26:01.527984 32723 leader_election.cc:341] T 127e6bec99db4dba8ba6ae51ea11e36d P ad1b6c08928b446ca0b8539b56223f57 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer eedca80ed76e4368a624213c9faed862 (127.17.53.66:43527): Not found: Tablet not found: 127e6bec99db4dba8ba6ae51ea11e36d
I20260929 20:26:01.528090 32723 leader_election.cc:302] T 127e6bec99db4dba8ba6ae51ea11e36d P ad1b6c08928b446ca0b8539b56223f57 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ad1b6c08928b446ca0b8539b56223f57; no voters: 77871fa252864a14a60aefdf7b95e58d, eedca80ed76e4368a624213c9faed862
I20260929 20:26:01.528323 647 raft_consensus.cc:2726] T 127e6bec99db4dba8ba6ae51ea11e36d P ad1b6c08928b446ca0b8539b56223f57 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 20:26:01.535079 648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:26:01.534249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:37213 (local address 127.17.53.65:47583)
0929 20:26:01.534725 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0929 20:26:01.534827 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.53.126:37213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":208,"threads_started":1}
I20260929 20:26:01.986922 649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:26:01.986182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:37213 (local address 127.17.53.66:36453)
0929 20:26:01.986609 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0929 20:26:01.986693 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.53.126:37213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":219,"threads_started":1}
I20260929 20:26:02.452584 650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:26:02.451877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:37213 (local address 127.17.53.67:52253)
0929 20:26:02.452341 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0929 20:26:02.452423 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.53.126:37213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":222,"threads_started":1}
I20260929 20:26:02.536751 651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:26:02.536032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:37213 (local address 127.17.53.65:56405)
0929 20:26:02.536482 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0929 20:26:02.536562 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.53.126:37213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":235,"threads_started":1}
I20260929 20:26:02.988523 652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:26:02.987823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:37213 (local address 127.17.53.66:33669)
0929 20:26:02.988258 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0929 20:26:02.988346 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.53.126:37213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":327,"thread_start_us":218,"threads_started":1}
I20260929 20:26:03.454344 653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:26:03.453587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:37213 (local address 127.17.53.67:55125)
0929 20:26:03.454050 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0929 20:26:03.454132 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.53.126:37213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":363,"thread_start_us":252,"threads_started":1}
I20260929 20:26:03.538617 654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:26:03.537843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:37213 (local address 127.17.53.65:56091)
0929 20:26:03.538314 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0929 20:26:03.538415 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.53.126:37213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":247,"threads_started":1}
I20260929 20:26:03.644835 655 raft_consensus.cc:482] T 127e6bec99db4dba8ba6ae51ea11e36d P ad1b6c08928b446ca0b8539b56223f57 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 20:26:03.645077 655 raft_consensus.cc:504] T 127e6bec99db4dba8ba6ae51ea11e36d P ad1b6c08928b446ca0b8539b56223f57 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad1b6c08928b446ca0b8539b56223f57" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 36129 } } peers { permanent_uuid: "eedca80ed76e4368a624213c9faed862" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 43527 } } peers { permanent_uuid: "77871fa252864a14a60aefdf7b95e58d" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 39129 } }
I20260929 20:26:03.645758 655 leader_election.cc:288] T 127e6bec99db4dba8ba6ae51ea11e36d P ad1b6c08928b446ca0b8539b56223f57 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers eedca80ed76e4368a624213c9faed862 (127.17.53.66:43527), 77871fa252864a14a60aefdf7b95e58d (127.17.53.67:39129)
W20260929 20:26:03.646780 32726 leader_election.cc:341] T 127e6bec99db4dba8ba6ae51ea11e36d P ad1b6c08928b446ca0b8539b56223f57 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 77871fa252864a14a60aefdf7b95e58d (127.17.53.67:39129): Not found: Tablet not found: 127e6bec99db4dba8ba6ae51ea11e36d
W20260929 20:26:03.647018 32723 leader_election.cc:341] T 127e6bec99db4dba8ba6ae51ea11e36d P ad1b6c08928b446ca0b8539b56223f57 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer eedca80ed76e4368a624213c9faed862 (127.17.53.66:43527): Not found: Tablet not found: 127e6bec99db4dba8ba6ae51ea11e36d
I20260929 20:26:03.647128 32723 leader_election.cc:302] T 127e6bec99db4dba8ba6ae51ea11e36d P ad1b6c08928b446ca0b8539b56223f57 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ad1b6c08928b446ca0b8539b56223f57; no voters: 77871fa252864a14a60aefdf7b95e58d, eedca80ed76e4368a624213c9faed862
I20260929 20:26:03.647349 655 raft_consensus.cc:2726] T 127e6bec99db4dba8ba6ae51ea11e36d P ad1b6c08928b446ca0b8539b56223f57 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 20:26:03.990268 656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:26:03.989564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:37213 (local address 127.17.53.66:50675)
0929 20:26:03.989994 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0929 20:26:03.990080 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.53.126:37213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":331,"thread_start_us":229,"threads_started":1}
I20260929 20:26:04.455976 657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:26:04.455253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:37213 (local address 127.17.53.67:36499)
0929 20:26:04.455718 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0929 20:26:04.455807 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.53.126:37213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":362,"thread_start_us":243,"threads_started":1}
I20260929 20:26:04.540501 658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:26:04.539696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:37213 (local address 127.17.53.65:39099)
0929 20:26:04.540246 (+ 550us) negotiation.cc:107] Waiting for socket to connect
0929 20:26:04.540327 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.53.126:37213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":430,"thread_start_us":338,"threads_started":1}
I20260929 20:26:04.992225 659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:26:04.991412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:37213 (local address 127.17.53.66:33377)
0929 20:26:04.991893 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0929 20:26:04.991990 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.53.126:37213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":247,"threads_started":1}
I20260929 20:26:05.457986 660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:26:05.457002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:37213 (local address 127.17.53.67:46643)
0929 20:26:05.457624 (+ 622us) negotiation.cc:107] Waiting for socket to connect
0929 20:26:05.457747 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.53.126:37213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":514,"thread_start_us":296,"threads_started":1}
I20260929 20:26:05.542415 661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:26:05.541515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:37213 (local address 127.17.53.65:38517)
0929 20:26:05.542013 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0929 20:26:05.542184 (+ 171us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.53.126:37213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":268,"threads_started":1}
I20260929 20:26:05.994040 662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:26:05.993195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:37213 (local address 127.17.53.66:54615)
0929 20:26:05.993756 (+ 561us) negotiation.cc:107] Waiting for socket to connect
0929 20:26:05.993852 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.53.126:37213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":240,"threads_started":1}
I20260929 20:26:06.459872 663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:26:06.459061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:37213 (local address 127.17.53.67:52227)
0929 20:26:06.459593 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0929 20:26:06.459691 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.53.126:37213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":430,"thread_start_us":229,"threads_started":1}
I20260929 20:26:06.544342 664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:26:06.543512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:37213 (local address 127.17.53.65:53181)
0929 20:26:06.543998 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0929 20:26:06.544172 (+ 174us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.53.126:37213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":386,"thread_start_us":265,"threads_started":1}
I20260929 20:26:06.995874 665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:26:06.995046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:37213 (local address 127.17.53.66:36613)
0929 20:26:06.995484 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0929 20:26:06.995620 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.53.126:37213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":333,"thread_start_us":210,"threads_started":1}
I20260929 20:26:07.461611 666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:26:07.460880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:37213 (local address 127.17.53.67:58651)
0929 20:26:07.461340 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0929 20:26:07.461427 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.53.126:37213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":353,"thread_start_us":242,"threads_started":1}
I20260929 20:26:07.546092 667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:26:07.545330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:37213 (local address 127.17.53.65:39253)
0929 20:26:07.545809 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0929 20:26:07.545912 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.53.126:37213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":382,"thread_start_us":232,"threads_started":1}
I20260929 20:26:07.997599 668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:26:07.996859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:37213 (local address 127.17.53.66:54567)
0929 20:26:07.997339 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0929 20:26:07.997432 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.53.126:37213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":226,"threads_started":1}
I20260929 20:26:08.433387 669 raft_consensus.cc:482] T 127e6bec99db4dba8ba6ae51ea11e36d P ad1b6c08928b446ca0b8539b56223f57 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 20:26:08.433601 669 raft_consensus.cc:504] T 127e6bec99db4dba8ba6ae51ea11e36d P ad1b6c08928b446ca0b8539b56223f57 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad1b6c08928b446ca0b8539b56223f57" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 36129 } } peers { permanent_uuid: "eedca80ed76e4368a624213c9faed862" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 43527 } } peers { permanent_uuid: "77871fa252864a14a60aefdf7b95e58d" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 39129 } }
I20260929 20:26:08.434309 669 leader_election.cc:288] T 127e6bec99db4dba8ba6ae51ea11e36d P ad1b6c08928b446ca0b8539b56223f57 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers eedca80ed76e4368a624213c9faed862 (127.17.53.66:43527), 77871fa252864a14a60aefdf7b95e58d (127.17.53.67:39129)
W20260929 20:26:08.435293 32723 leader_election.cc:341] T 127e6bec99db4dba8ba6ae51ea11e36d P ad1b6c08928b446ca0b8539b56223f57 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer eedca80ed76e4368a624213c9faed862 (127.17.53.66:43527): Not found: Tablet not found: 127e6bec99db4dba8ba6ae51ea11e36d
W20260929 20:26:08.435482 32726 leader_election.cc:341] T 127e6bec99db4dba8ba6ae51ea11e36d P ad1b6c08928b446ca0b8539b56223f57 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 77871fa252864a14a60aefdf7b95e58d (127.17.53.67:39129): Not found: Tablet not found: 127e6bec99db4dba8ba6ae51ea11e36d
I20260929 20:26:08.435637 32726 leader_election.cc:302] T 127e6bec99db4dba8ba6ae51ea11e36d P ad1b6c08928b446ca0b8539b56223f57 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ad1b6c08928b446ca0b8539b56223f57; no voters: 77871fa252864a14a60aefdf7b95e58d, eedca80ed76e4368a624213c9faed862
I20260929 20:26:08.435942 669 raft_consensus.cc:2726] T 127e6bec99db4dba8ba6ae51ea11e36d P ad1b6c08928b446ca0b8539b56223f57 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 20:26:08.463584 670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:26:08.462519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:37213 (local address 127.17.53.67:34965)
0929 20:26:08.463202 (+ 683us) negotiation.cc:107] Waiting for socket to connect
0929 20:26:08.463370 (+ 168us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.53.126:37213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":251,"threads_started":1}
I20260929 20:26:08.548233 671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:26:08.547288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:37213 (local address 127.17.53.65:58893)
0929 20:26:08.547851 (+ 563us) negotiation.cc:107] Waiting for socket to connect
0929 20:26:08.548009 (+ 158us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.53.126:37213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":318,"threads_started":1}
I20260929 20:26:08.999315 672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:26:08.998543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:37213 (local address 127.17.53.66:58361)
0929 20:26:08.999042 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0929 20:26:08.999128 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.53.126:37213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":402,"thread_start_us":259,"threads_started":1}
I20260929 20:26:09.465278 673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:26:09.464546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:37213 (local address 127.17.53.67:35741)
0929 20:26:09.465016 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0929 20:26:09.465102 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.53.126:37213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":368,"thread_start_us":254,"threads_started":1}
I20260929 20:26:09.549881 674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:26:09.549169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:37213 (local address 127.17.53.65:53365)
0929 20:26:09.549611 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0929 20:26:09.549700 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.53.126:37213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":338,"thread_start_us":237,"threads_started":1}
I20260929 20:26:10.001220 675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:26:10.000332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:37213 (local address 127.17.53.66:35367)
0929 20:26:10.000942 (+ 610us) negotiation.cc:107] Waiting for socket to connect
0929 20:26:10.001034 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.53.126:37213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":306,"threads_started":1}
I20260929 20:26:10.467044 676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:26:10.466255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:37213 (local address 127.17.53.67:36733)
0929 20:26:10.466729 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0929 20:26:10.466819 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.53.126:37213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":233,"threads_started":1}
I20260929 20:26:10.551855 677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:26:10.550982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:37213 (local address 127.17.53.65:56009)
0929 20:26:10.551539 (+ 557us) negotiation.cc:107] Waiting for socket to connect
0929 20:26:10.551645 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.53.126:37213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":339,"threads_started":1}
I20260929 20:26:11.003095 678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:26:11.002308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:37213 (local address 127.17.53.66:53139)
0929 20:26:11.002777 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0929 20:26:11.002920 (+ 143us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.53.126:37213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":237,"threads_started":1}
I20260929 20:26:11.468808 679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:26:11.468122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:37213 (local address 127.17.53.67:45279)
0929 20:26:11.468558 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0929 20:26:11.468637 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.53.126:37213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":230,"threads_started":1}
I20260929 20:26:11.553650 680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:26:11.552858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:37213 (local address 127.17.53.65:49895)
0929 20:26:11.553335 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0929 20:26:11.553442 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.53.126:37213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":371,"thread_start_us":254,"threads_started":1}
I20260929 20:26:12.004801 681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:26:12.004033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:37213 (local address 127.17.53.66:39665)
0929 20:26:12.004497 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0929 20:26:12.004626 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.53.126:37213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":248,"threads_started":1}
I20260929 20:26:12.470935 682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:26:12.469807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:37213 (local address 127.17.53.67:42157)
0929 20:26:12.470521 (+ 714us) negotiation.cc:107] Waiting for socket to connect
0929 20:26:12.470681 (+ 160us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.53.126:37213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":605,"thread_start_us":488,"threads_started":1}
I20260929 20:26:12.555590 683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:26:12.554664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:37213 (local address 127.17.53.65:44809)
0929 20:26:12.555303 (+ 639us) negotiation.cc:107] Waiting for socket to connect
0929 20:26:12.555390 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.53.126:37213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":515,"thread_start_us":412,"threads_started":1}
I20260929 20:26:13.006723 684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:26:13.005860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:37213 (local address 127.17.53.66:46661)
0929 20:26:13.006390 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0929 20:26:13.006499 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.53.126:37213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":417,"thread_start_us":313,"threads_started":1}
I20260929 20:26:13.174496 685 raft_consensus.cc:482] T 127e6bec99db4dba8ba6ae51ea11e36d P ad1b6c08928b446ca0b8539b56223f57 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 20:26:13.174712 685 raft_consensus.cc:504] T 127e6bec99db4dba8ba6ae51ea11e36d P ad1b6c08928b446ca0b8539b56223f57 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ad1b6c08928b446ca0b8539b56223f57" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 36129 } } peers { permanent_uuid: "eedca80ed76e4368a624213c9faed862" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 43527 } } peers { permanent_uuid: "77871fa252864a14a60aefdf7b95e58d" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 39129 } }
I20260929 20:26:13.175414 685 leader_election.cc:288] T 127e6bec99db4dba8ba6ae51ea11e36d P ad1b6c08928b446ca0b8539b56223f57 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers eedca80ed76e4368a624213c9faed862 (127.17.53.66:43527), 77871fa252864a14a60aefdf7b95e58d (127.17.53.67:39129)
W20260929 20:26:13.176477 32723 leader_election.cc:341] T 127e6bec99db4dba8ba6ae51ea11e36d P ad1b6c08928b446ca0b8539b56223f57 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer eedca80ed76e4368a624213c9faed862 (127.17.53.66:43527): Not found: Tablet not found: 127e6bec99db4dba8ba6ae51ea11e36d
W20260929 20:26:13.176715 32726 leader_election.cc:341] T 127e6bec99db4dba8ba6ae51ea11e36d P ad1b6c08928b446ca0b8539b56223f57 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 77871fa252864a14a60aefdf7b95e58d (127.17.53.67:39129): Not found: Tablet not found: 127e6bec99db4dba8ba6ae51ea11e36d
I20260929 20:26:13.176826 32726 leader_election.cc:302] T 127e6bec99db4dba8ba6ae51ea11e36d P ad1b6c08928b446ca0b8539b56223f57 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ad1b6c08928b446ca0b8539b56223f57; no voters: 77871fa252864a14a60aefdf7b95e58d, eedca80ed76e4368a624213c9faed862
I20260929 20:26:13.177047 685 raft_consensus.cc:2726] T 127e6bec99db4dba8ba6ae51ea11e36d P ad1b6c08928b446ca0b8539b56223f57 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 20:26:13.472690 686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:26:13.471916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:37213 (local address 127.17.53.67:48829)
0929 20:26:13.472387 (+ 471us) negotiation.cc:107] Waiting for socket to connect
0929 20:26:13.472485 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.53.126:37213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":368,"thread_start_us":262,"threads_started":1}
I20260929 20:26:13.557945 687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:26:13.556897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:37213 (local address 127.17.53.65:41613)
0929 20:26:13.557617 (+ 720us) negotiation.cc:107] Waiting for socket to connect
0929 20:26:13.557767 (+ 150us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.53.126:37213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":591,"thread_start_us":460,"threads_started":1}
I20260929 20:26:14.008816 688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:26:14.008039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:37213 (local address 127.17.53.66:41355)
0929 20:26:14.008535 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0929 20:26:14.008632 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.53.126:37213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":240,"threads_started":1}
I20260929 20:26:14.457206 17621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu pid 32673
I20260929 20:26:14.472740 689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:26:14.458500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53332 (local address 127.17.53.65:36129)
0929 20:26:14.459260 (+ 760us) server_negotiation.cc:206] Beginning negotiation
0929 20:26:14.459270 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 20:26:14.459304 (+ 34us) server_negotiation.cc:407] Connection header received
0929 20:26:14.459398 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:26:14.459407 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:26:14.459713 (+ 306us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:26:14.459906 (+ 193us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:26:14.461359 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:26:14.462335 (+ 976us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:26:14.463652 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:26:14.464218 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:26:14.464362 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:26:14.467108 (+ 2746us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:26:14.467119 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:26:14.467133 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:26:14.467176 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:26:14.469765 (+ 2589us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:26:14.470488 (+ 723us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:26:14.470501 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:26:14.470503 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:26:14.470649 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:26:14.471087 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:26:14.471092 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:26:14.471096 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:26:14.471461 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:26:14.471768 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0929 20:26:14.472123 (+ 355us) server_negotiation.cc:299] Negotiation successful
0929 20:26:14.472414 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":543,"thread_start_us":384,"threads_started":1}
I20260929 20:26:14.474052 364 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:26:14.474591 690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:26:14.473818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:37213 (local address 127.17.53.67:39045)
0929 20:26:14.474331 (+ 513us) negotiation.cc:107] Waiting for socket to connect
0929 20:26:14.474446 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.53.126:37213: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":390,"thread_start_us":229,"threads_started":1}
I20260929 20:26:14.564898 17621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu with pid 32673
I20260929 20:26:14.582939 17621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu pid 372
I20260929 20:26:14.598098 693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:26:14.583707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40324 (local address 127.17.53.66:43527)
0929 20:26:14.584224 (+ 517us) server_negotiation.cc:206] Beginning negotiation
0929 20:26:14.584228 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:26:14.584244 (+ 16us) server_negotiation.cc:407] Connection header received
0929 20:26:14.584296 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:26:14.584299 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:26:14.584355 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:26:14.584471 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:26:14.586232 (+ 1761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:26:14.587277 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:26:14.588681 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:26:14.589160 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:26:14.589309 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:26:14.592354 (+ 3045us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:26:14.592360 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:26:14.592363 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:26:14.592401 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:26:14.595313 (+ 2912us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:26:14.596031 (+ 718us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:26:14.596038 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:26:14.596041 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:26:14.596190 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:26:14.596580 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:26:14.596596 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:26:14.596598 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:26:14.596877 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:26:14.597178 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0929 20:26:14.597488 (+ 310us) server_negotiation.cc:299] Negotiation successful
0929 20:26:14.597721 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":371,"thread_start_us":246,"threads_started":1}
I20260929 20:26:14.598783 497 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:26:14.684449 17621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu with pid 372
I20260929 20:26:14.703127 17621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu pid 505
I20260929 20:26:14.717818 695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:26:14.704336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44640 (local address 127.17.53.67:39129)
0929 20:26:14.704932 (+ 596us) server_negotiation.cc:206] Beginning negotiation
0929 20:26:14.704936 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:26:14.704952 (+ 16us) server_negotiation.cc:407] Connection header received
0929 20:26:14.705007 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:26:14.705010 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:26:14.705077 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:26:14.705174 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:26:14.706825 (+ 1651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:26:14.707676 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:26:14.709197 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:26:14.709696 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:26:14.709795 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:26:14.712558 (+ 2763us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:26:14.712563 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:26:14.712566 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:26:14.712601 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:26:14.715467 (+ 2866us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:26:14.716114 (+ 647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:26:14.716117 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:26:14.716120 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:26:14.716290 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:26:14.716640 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:26:14.716645 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:26:14.716648 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:26:14.716897 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:26:14.717091 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0929 20:26:14.717354 (+ 263us) server_negotiation.cc:299] Negotiation successful
0929 20:26:14.717515 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":477,"thread_start_us":340,"threads_started":1}
I20260929 20:26:14.718524 628 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:26:14.801292 17621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu with pid 505
2026-09-29T20:26:14Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (24487 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713339161492-17621-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 20:26:14 dist-test-slave-zbnc krb5kdc[699](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 20:26:14 dist-test-slave-zbnc krb5kdc[699](info): set up 2 sockets
Sep 29 20:26:14 dist-test-slave-zbnc krb5kdc[699](info): commencing operation
krb5kdc: starting...
W20260929 20:26:15.920975 17621 mini_kdc.cc:121] Time spent starting KDC: real 1.079s user 0.004s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713339161492-17621-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713339161492-17621-0/krb5kdc/test-user.keytab.
Sep 29 20:26:16 dist-test-slave-zbnc krb5kdc[699](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713576, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T20:26:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T20:26:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.17.53.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.53.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.53.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.126.keytab.
Entry for principal kudu/127.17.53.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.126.keytab.
I20260929 20:26:16.166399 17621 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
/tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713339161492-17621-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713339161492-17621-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713339161492-17621-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713339161492-17621-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.53.126:39625
--webserver_interface=127.17.53.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.126.keytab
--principal=kudu/127.17.53.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.53.84:36709
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.53.126:39625
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713339161492-17621-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713339161492-17621-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713339161492-17621-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:26:16.498729 712 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:26:16.499110 712 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:26:16.499301 712 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:26:16.507170 712 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260929 20:26:16.507272 712 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260929 20:26:16.507359 712 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260929 20:26:16.507448 712 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:26:16.507535 712 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:26:16.507616 712 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:26:16.507661 712 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:26:16.507716 712 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:26:16.517859 712 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.53.84:36709
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713339161492-17621-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713339161492-17621-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.17.53.126:39625
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.126.keytab
--principal=kudu/127.17.53.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.53.126:39625
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713339161492-17621-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.53.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713339161492-17621-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:26:16.519174 712 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:26:16.520519 712 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:26:16.533546 718 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:26:16.533568 720 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:26:16.533810 717 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:26:16.534406 712 server_base.cc:1034] running on GCE node
I20260929 20:26:16.535396 712 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:26:16.536839 712 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:26:16.538162 712 hybrid_clock.cc:648] HybridClock initialized: now 1790713576538103 us; error 35 us; skew 500 ppm
I20260929 20:26:16.538574 712 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:26:16 dist-test-slave-zbnc krb5kdc[699](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713576, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.53.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:26:16.542038 712 init.cc:377] Logged in from keytab as kudu/127.17.53.126@KRBTEST.COM (short username kudu)
I20260929 20:26:16.543531 712 webserver.cc:460] Webserver started at http://127.17.53.126:42051/ using document root <none> and password file <none>
I20260929 20:26:16.544201 712 fs_manager.cc:359] Metadata directory not provided
I20260929 20:26:16.544286 712 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:26:16.544554 712 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:26:16.547607 712 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713339161492-17621-0/minicluster-data/master-0/data/instance:
uuid: "e5f2bfb42723428996a3b2e3bb46f683"
format_stamp: "Formatted at 2026-09-29 20:26:16 on dist-test-slave-zbnc"
server_key: "6f9bd922439cf9858a42a8d6ec9ce281"
server_key_iv: "dcf1d40a474d3e29ab4ba8f68c56b511"
server_key_version: "encryptionkey@0"
I20260929 20:26:16.548437 712 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713339161492-17621-0/minicluster-data/master-0/wal/instance:
uuid: "e5f2bfb42723428996a3b2e3bb46f683"
format_stamp: "Formatted at 2026-09-29 20:26:16 on dist-test-slave-zbnc"
server_key: "6f9bd922439cf9858a42a8d6ec9ce281"
server_key_iv: "dcf1d40a474d3e29ab4ba8f68c56b511"
server_key_version: "encryptionkey@0"
I20260929 20:26:16.553611 712 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 20:26:16.557233 727 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:26:16.558741 712 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 20:26:16.558938 712 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713339161492-17621-0/minicluster-data/master-0/data,/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713339161492-17621-0/minicluster-data/master-0/wal
uuid: "e5f2bfb42723428996a3b2e3bb46f683"
format_stamp: "Formatted at 2026-09-29 20:26:16 on dist-test-slave-zbnc"
server_key: "6f9bd922439cf9858a42a8d6ec9ce281"
server_key_iv: "dcf1d40a474d3e29ab4ba8f68c56b511"
server_key_version: "encryptionkey@0"
I20260929 20:26:16.559163 712 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713339161492-17621-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713339161492-17621-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713339161492-17621-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:26:16.577914 712 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:26:16.582412 712 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:26:16.582700 712 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:26:16.603091 712 rpc_server.cc:307] RPC server started. Bound to: 127.17.53.126:39625
I20260929 20:26:16.603070 778 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.53.126:39625 every 8 connection(s)
I20260929 20:26:16.604534 712 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713339161492-17621-0/minicluster-data/master-0/data/info.pb
I20260929 20:26:16.608439 779 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:26:16.609707 17621 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu as pid 712
I20260929 20:26:16.609926 17621 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713339161492-17621-0/minicluster-data/master-0/wal/instance
I20260929 20:26:16.610756 17621 external_mini_cluster.cc:1428] Setting key 45b1f30869b6d3afa06882fcc6b6c8ab
I20260929 20:26:16.623324 779 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e5f2bfb42723428996a3b2e3bb46f683: Bootstrap starting.
Sep 29 20:26:16 dist-test-slave-zbnc krb5kdc[699](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713576, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.53.126@KRBTEST.COM
I20260929 20:26:16.627245 779 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e5f2bfb42723428996a3b2e3bb46f683: Neither blocks nor log segments found. Creating new log.
I20260929 20:26:16.628353 779 log.cc:824] T 00000000000000000000000000000000 P e5f2bfb42723428996a3b2e3bb46f683: Log is configured to *not* fsync() on all Append() calls
I20260929 20:26:16.631879 779 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e5f2bfb42723428996a3b2e3bb46f683: No bootstrap required, opened a new log
I20260929 20:26:16.636227 782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:26:16.612904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49890 (local address 127.17.53.126:39625)
0929 20:26:16.613714 (+ 810us) server_negotiation.cc:206] Beginning negotiation
0929 20:26:16.613728 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 20:26:16.613775 (+ 47us) server_negotiation.cc:407] Connection header received
0929 20:26:16.615452 (+ 1677us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:26:16.615474 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:26:16.615863 (+ 389us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:26:16.616194 (+ 331us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:26:16.618027 (+ 1833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:26:16.619436 (+ 1409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:26:16.621361 (+ 1925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:26:16.622096 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:26:16.622252 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:26:16.627535 (+ 5283us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:26:16.627543 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:26:16.627558 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:26:16.627603 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:26:16.631188 (+ 3585us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:26:16.632160 (+ 972us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:26:16.632170 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:26:16.632182 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:26:16.632418 (+ 236us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:26:16.632847 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:26:16.632852 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:26:16.632855 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:26:16.633562 (+ 707us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:26:16.633886 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0929 20:26:16.634273 (+ 387us) server_negotiation.cc:299] Negotiation successful
0929 20:26:16.634760 (+ 487us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":594,"thread_start_us":382,"threads_started":1}
I20260929 20:26:16.641510 779 raft_consensus.cc:348] T 00000000000000000000000000000000 P e5f2bfb42723428996a3b2e3bb46f683 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5f2bfb42723428996a3b2e3bb46f683" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 39625 } }
I20260929 20:26:16.641896 779 raft_consensus.cc:374] T 00000000000000000000000000000000 P e5f2bfb42723428996a3b2e3bb46f683 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:26:16.642016 779 raft_consensus.cc:729] T 00000000000000000000000000000000 P e5f2bfb42723428996a3b2e3bb46f683 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5f2bfb42723428996a3b2e3bb46f683, State: Initialized, Role: FOLLOWER
I20260929 20:26:16.642637 779 consensus_queue.cc:260] T 00000000000000000000000000000000 P e5f2bfb42723428996a3b2e3bb46f683 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5f2bfb42723428996a3b2e3bb46f683" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 39625 } }
I20260929 20:26:16.642844 779 raft_consensus.cc:388] T 00000000000000000000000000000000 P e5f2bfb42723428996a3b2e3bb46f683 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 20:26:16.643006 779 raft_consensus.cc:482] T 00000000000000000000000000000000 P e5f2bfb42723428996a3b2e3bb46f683 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 20:26:16.643123 779 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e5f2bfb42723428996a3b2e3bb46f683 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:26:16.645228 779 raft_consensus.cc:504] T 00000000000000000000000000000000 P e5f2bfb42723428996a3b2e3bb46f683 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5f2bfb42723428996a3b2e3bb46f683" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 39625 } }
I20260929 20:26:16.645650 779 leader_election.cc:302] T 00000000000000000000000000000000 P e5f2bfb42723428996a3b2e3bb46f683 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e5f2bfb42723428996a3b2e3bb46f683; no voters:
I20260929 20:26:16.646279 779 leader_election.cc:288] T 00000000000000000000000000000000 P e5f2bfb42723428996a3b2e3bb46f683 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 20:26:16.646373 784 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e5f2bfb42723428996a3b2e3bb46f683 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:26:16.647161 784 raft_consensus.cc:686] T 00000000000000000000000000000000 P e5f2bfb42723428996a3b2e3bb46f683 [term 1 LEADER]: Becoming Leader. State: Replica: e5f2bfb42723428996a3b2e3bb46f683, State: Running, Role: LEADER
I20260929 20:26:16.647783 784 consensus_queue.cc:237] T 00000000000000000000000000000000 P e5f2bfb42723428996a3b2e3bb46f683 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5f2bfb42723428996a3b2e3bb46f683" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 39625 } }
I20260929 20:26:16.648788 779 sys_catalog.cc:564] T 00000000000000000000000000000000 P e5f2bfb42723428996a3b2e3bb46f683 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:26:16.650722 785 sys_catalog.cc:455] T 00000000000000000000000000000000 P e5f2bfb42723428996a3b2e3bb46f683 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e5f2bfb42723428996a3b2e3bb46f683. Latest consensus state: current_term: 1 leader_uuid: "e5f2bfb42723428996a3b2e3bb46f683" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5f2bfb42723428996a3b2e3bb46f683" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 39625 } } }
I20260929 20:26:16.651201 785 sys_catalog.cc:458] T 00000000000000000000000000000000 P e5f2bfb42723428996a3b2e3bb46f683 [sys.catalog]: This master's current role is: LEADER
I20260929 20:26:16.651208 786 sys_catalog.cc:455] T 00000000000000000000000000000000 P e5f2bfb42723428996a3b2e3bb46f683 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e5f2bfb42723428996a3b2e3bb46f683" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5f2bfb42723428996a3b2e3bb46f683" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 39625 } } }
I20260929 20:26:16.653079 786 sys_catalog.cc:458] T 00000000000000000000000000000000 P e5f2bfb42723428996a3b2e3bb46f683 [sys.catalog]: This master's current role is: LEADER
I20260929 20:26:16.653244 789 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:26:16.657341 789 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:26:16.667526 789 catalog_manager.cc:1347] Generated new cluster ID: 440651dfe5e2435689788eaff7deaa18
I20260929 20:26:16.667652 789 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:26:16.678030 789 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 20:26:16.679491 789 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:26:16.696643 789 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e5f2bfb42723428996a3b2e3bb46f683: Generated new TSK 0
I20260929 20:26:16.697695 789 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.17.53.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.53.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.53.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.65.keytab.
Entry for principal kudu/127.17.53.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.65.keytab.
I20260929 20:26:16.763391 17621 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
/tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713339161492-17621-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713339161492-17621-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713339161492-17621-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713339161492-17621-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.53.65:0
--local_ip_for_outbound_sockets=127.17.53.65
--webserver_interface=127.17.53.65
--webserver_port=0
--tserver_master_addrs=127.17.53.126:39625
--keytab_file=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.65.keytab
--principal=kudu/127.17.53.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.53.84:36709
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713339161492-17621-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713339161492-17621-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713339161492-17621-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:26:17.092851 805 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:26:17.093133 805 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:26:17.093228 805 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:26:17.100368 805 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:26:17.100487 805 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:26:17.100600 805 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.53.65
I20260929 20:26:17.110504 805 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.53.84:36709
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713339161492-17621-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713339161492-17621-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.65.keytab
--principal=kudu/127.17.53.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.53.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713339161492-17621-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.17.53.65
--webserver_port=0
--tserver_master_addrs=127.17.53.126:39625
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.53.65
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713339161492-17621-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:26:17.111881 805 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:26:17.113196 805 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:26:17.127117 810 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:26:17.127094 811 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:26:17.127594 813 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:26:17.128053 805 server_base.cc:1034] running on GCE node
I20260929 20:26:17.128683 805 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:26:17.129609 805 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:26:17.130848 805 hybrid_clock.cc:648] HybridClock initialized: now 1790713577130795 us; error 47 us; skew 500 ppm
I20260929 20:26:17.131314 805 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:26:17 dist-test-slave-zbnc krb5kdc[699](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713577, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.53.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:26:17.135102 805 init.cc:377] Logged in from keytab as kudu/127.17.53.65@KRBTEST.COM (short username kudu)
I20260929 20:26:17.136672 805 webserver.cc:460] Webserver started at http://127.17.53.65:33787/ using document root <none> and password file <none>
I20260929 20:26:17.137393 805 fs_manager.cc:359] Metadata directory not provided
I20260929 20:26:17.137544 805 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:26:17.137872 805 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:26:17.141040 805 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713339161492-17621-0/minicluster-data/ts-0/data/instance:
uuid: "6e489a4674b840c4b61f170102698b7d"
format_stamp: "Formatted at 2026-09-29 20:26:17 on dist-test-slave-zbnc"
server_key: "5684d05b4cfcfc1fb7d6fa54ad8e7c7b"
server_key_iv: "c257d9544ebce464fb7d5810b92342c8"
server_key_version: "encryptionkey@0"
I20260929 20:26:17.141896 805 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713339161492-17621-0/minicluster-data/ts-0/wal/instance:
uuid: "6e489a4674b840c4b61f170102698b7d"
format_stamp: "Formatted at 2026-09-29 20:26:17 on dist-test-slave-zbnc"
server_key: "5684d05b4cfcfc1fb7d6fa54ad8e7c7b"
server_key_iv: "c257d9544ebce464fb7d5810b92342c8"
server_key_version: "encryptionkey@0"
I20260929 20:26:17.147305 805 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260929 20:26:17.150758 820 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:26:17.152139 805 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 20:26:17.152305 805 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713339161492-17621-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713339161492-17621-0/minicluster-data/ts-0/wal
uuid: "6e489a4674b840c4b61f170102698b7d"
format_stamp: "Formatted at 2026-09-29 20:26:17 on dist-test-slave-zbnc"
server_key: "5684d05b4cfcfc1fb7d6fa54ad8e7c7b"
server_key_iv: "c257d9544ebce464fb7d5810b92342c8"
server_key_version: "encryptionkey@0"
I20260929 20:26:17.152503 805 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713339161492-17621-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713339161492-17621-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713339161492-17621-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:26:17.180256 805 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:26:17.184726 805 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:26:17.185035 805 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:26:17.186088 805 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:26:17.187554 805 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:26:17.187671 805 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:26:17.187822 805 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:26:17.187897 805 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:26:17.221014 805 rpc_server.cc:307] RPC server started. Bound to: 127.17.53.65:43287
I20260929 20:26:17.221174 932 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.53.65:43287 every 8 connection(s)
I20260929 20:26:17.222332 805 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713339161492-17621-0/minicluster-data/ts-0/data/info.pb
I20260929 20:26:17.226539 17621 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu as pid 805
I20260929 20:26:17.226910 17621 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713339161492-17621-0/minicluster-data/ts-0/wal/instance
I20260929 20:26:17.227669 17621 external_mini_cluster.cc:1428] Setting key 7caefa7166d6d6359dfcd07e87a45651
Sep 29 20:26:17 dist-test-slave-zbnc krb5kdc[699](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713577, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.53.65@KRBTEST.COM for kudu/127.17.53.126@KRBTEST.COM
I20260929 20:26:17.246605 935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:26:17.225051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.53.65:49687 (local address 127.17.53.126:39625)
0929 20:26:17.225531 (+ 480us) server_negotiation.cc:206] Beginning negotiation
0929 20:26:17.225535 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:26:17.226523 (+ 988us) server_negotiation.cc:407] Connection header received
0929 20:26:17.228714 (+ 2191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:26:17.228720 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:26:17.228822 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:26:17.228946 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:26:17.230615 (+ 1669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:26:17.231416 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:26:17.232957 (+ 1541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:26:17.233628 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:26:17.233795 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:26:17.240908 (+ 7113us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:26:17.240913 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:26:17.240916 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:26:17.240949 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:26:17.243596 (+ 2647us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:26:17.244313 (+ 717us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:26:17.244320 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:26:17.244323 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:26:17.244441 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:26:17.244865 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:26:17.244873 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:26:17.244877 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:26:17.245144 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:26:17.245307 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0929 20:26:17.245999 (+ 692us) server_negotiation.cc:299] Negotiation successful
0929 20:26:17.246234 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":373,"thread_start_us":246,"threads_started":1}
I20260929 20:26:17.247606 936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:26:17.225643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:39625 (local address 127.17.53.65:49687)
0929 20:26:17.226357 (+ 714us) negotiation.cc:107] Waiting for socket to connect
0929 20:26:17.226405 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0929 20:26:17.228416 (+ 2011us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:26:17.229205 (+ 789us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:26:17.229214 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:26:17.229867 (+ 653us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:26:17.230449 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:26:17.230465 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:26:17.231567 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:26:17.231575 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:26:17.232802 (+ 1227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:26:17.232818 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:26:17.233881 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:26:17.233890 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:26:17.234174 (+ 284us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:26:17.238134 (+ 3960us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:26:17.238160 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:26:17.240630 (+ 2470us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:26:17.243754 (+ 3124us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:26:17.243764 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:26:17.243781 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:26:17.244191 (+ 410us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:26:17.244553 (+ 362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:26:17.244558 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:26:17.244562 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:26:17.244753 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:26:17.245278 (+ 525us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:26:17.245286 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:26:17.245628 (+ 342us) client_negotiation.cc:769] Sending connection context
0929 20:26:17.245906 (+ 278us) client_negotiation.cc:240] Negotiation successful
0929 20:26:17.246437 (+ 531us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":492,"thread_start_us":309,"threads_started":1}
I20260929 20:26:17.249241 933 heartbeater.cc:347] Connected to a master server at 127.17.53.126:39625
I20260929 20:26:17.249594 933 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.17.53.66@KRBTEST.COM; defaulting to no policy
I20260929 20:26:17.250496 933 heartbeater.cc:511] Master 127.17.53.126:39625 requested a full tablet report, sending...
I20260929 20:26:17.252794 744 ts_manager.cc:194] Registered new tserver with Master: 6e489a4674b840c4b61f170102698b7d (127.17.53.65:43287)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.53.66@KRBTEST.COM" created.
I20260929 20:26:17.254316 744 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.53.65@KRBTEST.COM'} at 127.17.53.65:49687
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.53.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.66.keytab.
Entry for principal kudu/127.17.53.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.66.keytab.
I20260929 20:26:17.282771 17621 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
/tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713339161492-17621-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713339161492-17621-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713339161492-17621-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713339161492-17621-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.53.66:0
--local_ip_for_outbound_sockets=127.17.53.66
--webserver_interface=127.17.53.66
--webserver_port=0
--tserver_master_addrs=127.17.53.126:39625
--keytab_file=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.66.keytab
--principal=kudu/127.17.53.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.53.84:36709
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713339161492-17621-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713339161492-17621-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713339161492-17621-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:26:17.611546 939 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:26:17.611862 939 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:26:17.612047 939 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:26:17.619578 939 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:26:17.619724 939 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:26:17.619910 939 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.53.66
I20260929 20:26:17.629801 939 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.53.84:36709
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713339161492-17621-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713339161492-17621-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.66.keytab
--principal=kudu/127.17.53.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.53.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713339161492-17621-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.17.53.66
--webserver_port=0
--tserver_master_addrs=127.17.53.126:39625
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.53.66
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713339161492-17621-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:26:17.631197 939 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:26:17.632526 939 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:26:17.646639 944 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:26:17.646721 945 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:26:17.646816 947 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:26:17.647426 939 server_base.cc:1034] running on GCE node
I20260929 20:26:17.648022 939 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:26:17.648958 939 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:26:17.650216 939 hybrid_clock.cc:648] HybridClock initialized: now 1790713577650114 us; error 76 us; skew 500 ppm
I20260929 20:26:17.650548 939 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:26:17 dist-test-slave-zbnc krb5kdc[699](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713577, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.53.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:26:17.653666 939 init.cc:377] Logged in from keytab as kudu/127.17.53.66@KRBTEST.COM (short username kudu)
I20260929 20:26:17.655329 939 webserver.cc:460] Webserver started at http://127.17.53.66:37051/ using document root <none> and password file <none>
I20260929 20:26:17.656261 939 fs_manager.cc:359] Metadata directory not provided
I20260929 20:26:17.656366 939 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:26:17.656617 939 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:26:17.659628 939 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713339161492-17621-0/minicluster-data/ts-1/data/instance:
uuid: "3b59fcbacafc43b088c6cacc664f6ccf"
format_stamp: "Formatted at 2026-09-29 20:26:17 on dist-test-slave-zbnc"
server_key: "79ab5d60f09934e4ae2ad1cda3c47a82"
server_key_iv: "a2995c358f36bfdcdfa0e549a65cbd2e"
server_key_version: "encryptionkey@0"
I20260929 20:26:17.660331 939 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713339161492-17621-0/minicluster-data/ts-1/wal/instance:
uuid: "3b59fcbacafc43b088c6cacc664f6ccf"
format_stamp: "Formatted at 2026-09-29 20:26:17 on dist-test-slave-zbnc"
server_key: "79ab5d60f09934e4ae2ad1cda3c47a82"
server_key_iv: "a2995c358f36bfdcdfa0e549a65cbd2e"
server_key_version: "encryptionkey@0"
I20260929 20:26:17.665539 939 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 20:26:17.668854 954 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:26:17.670145 939 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 20:26:17.670325 939 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713339161492-17621-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713339161492-17621-0/minicluster-data/ts-1/wal
uuid: "3b59fcbacafc43b088c6cacc664f6ccf"
format_stamp: "Formatted at 2026-09-29 20:26:17 on dist-test-slave-zbnc"
server_key: "79ab5d60f09934e4ae2ad1cda3c47a82"
server_key_iv: "a2995c358f36bfdcdfa0e549a65cbd2e"
server_key_version: "encryptionkey@0"
I20260929 20:26:17.670523 939 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713339161492-17621-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713339161492-17621-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713339161492-17621-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:26:17.691910 939 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:26:17.696355 939 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:26:17.696657 939 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:26:17.697662 939 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:26:17.699007 939 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:26:17.699112 939 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:26:17.699261 939 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:26:17.699334 939 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:26:17.732829 939 rpc_server.cc:307] RPC server started. Bound to: 127.17.53.66:33821
I20260929 20:26:17.732874 1066 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.53.66:33821 every 8 connection(s)
I20260929 20:26:17.734127 939 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713339161492-17621-0/minicluster-data/ts-1/data/info.pb
I20260929 20:26:17.735028 17621 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu as pid 939
I20260929 20:26:17.735229 17621 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713339161492-17621-0/minicluster-data/ts-1/wal/instance
I20260929 20:26:17.735854 17621 external_mini_cluster.cc:1428] Setting key 5381774adab31ece8400fbe789ee50a8
Sep 29 20:26:17 dist-test-slave-zbnc krb5kdc[699](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713577, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.53.66@KRBTEST.COM for kudu/127.17.53.126@KRBTEST.COM
I20260929 20:26:17.758476 935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:26:17.737079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.53.66:45813 (local address 127.17.53.126:39625)
0929 20:26:17.737254 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0929 20:26:17.737258 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:26:17.738792 (+ 1534us) server_negotiation.cc:407] Connection header received
0929 20:26:17.741747 (+ 2955us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:26:17.741755 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:26:17.741845 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:26:17.741949 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:26:17.744087 (+ 2138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:26:17.745099 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:26:17.747535 (+ 2436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:26:17.747983 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:26:17.748121 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:26:17.752193 (+ 4072us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:26:17.752197 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:26:17.752201 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:26:17.752241 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:26:17.754834 (+ 2593us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:26:17.755840 (+ 1006us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:26:17.755845 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:26:17.755847 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:26:17.755968 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:26:17.756545 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:26:17.756548 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:26:17.756550 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:26:17.756912 (+ 362us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:26:17.757026 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0929 20:26:17.757947 (+ 921us) server_negotiation.cc:299] Negotiation successful
0929 20:26:17.758123 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 20:26:17.759610 1069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:26:17.737707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:39625 (local address 127.17.53.66:45813)
0929 20:26:17.738613 (+ 906us) negotiation.cc:107] Waiting for socket to connect
0929 20:26:17.738668 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0929 20:26:17.741435 (+ 2767us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:26:17.742233 (+ 798us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:26:17.742248 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:26:17.743060 (+ 812us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:26:17.743905 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:26:17.743933 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:26:17.745250 (+ 1317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:26:17.745258 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:26:17.747386 (+ 2128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:26:17.747398 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:26:17.748228 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:26:17.748236 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:26:17.748575 (+ 339us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:26:17.749306 (+ 731us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:26:17.749333 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:26:17.751911 (+ 2578us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:26:17.755023 (+ 3112us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:26:17.755034 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:26:17.755055 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:26:17.755702 (+ 647us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:26:17.756081 (+ 379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:26:17.756088 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:26:17.756094 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:26:17.756355 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:26:17.757045 (+ 690us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:26:17.757055 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:26:17.757526 (+ 471us) client_negotiation.cc:769] Sending connection context
0929 20:26:17.757862 (+ 336us) client_negotiation.cc:240] Negotiation successful
0929 20:26:17.758448 (+ 586us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":606,"thread_start_us":365,"threads_started":1}
I20260929 20:26:17.761202 1067 heartbeater.cc:347] Connected to a master server at 127.17.53.126:39625
I20260929 20:26:17.761512 1067 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.17.53.67@KRBTEST.COM; defaulting to no policy
I20260929 20:26:17.762324 1067 heartbeater.cc:511] Master 127.17.53.126:39625 requested a full tablet report, sending...
I20260929 20:26:17.763890 744 ts_manager.cc:194] Registered new tserver with Master: 3b59fcbacafc43b088c6cacc664f6ccf (127.17.53.66:33821)
I20260929 20:26:17.764691 744 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.53.66@KRBTEST.COM'} at 127.17.53.66:45813
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.53.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.53.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.67.keytab.
Entry for principal kudu/127.17.53.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.67.keytab.
I20260929 20:26:17.792162 17621 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
/tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713339161492-17621-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713339161492-17621-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713339161492-17621-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713339161492-17621-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.53.67:0
--local_ip_for_outbound_sockets=127.17.53.67
--webserver_interface=127.17.53.67
--webserver_port=0
--tserver_master_addrs=127.17.53.126:39625
--keytab_file=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.67.keytab
--principal=kudu/127.17.53.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.53.84:36709
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713339161492-17621-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713339161492-17621-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713339161492-17621-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:26:18.117439 1072 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:26:18.117743 1072 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:26:18.117947 1072 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:26:18.125991 1072 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:26:18.126107 1072 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:26:18.126298 1072 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.53.67
I20260929 20:26:18.136664 1072 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.53.84:36709
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713339161492-17621-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713339161492-17621-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.67.keytab
--principal=kudu/127.17.53.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.53.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713339161492-17621-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.17.53.67
--webserver_port=0
--tserver_master_addrs=127.17.53.126:39625
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.53.67
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713339161492-17621-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:26:18.138096 1072 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:26:18.139549 1072 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:26:18.153792 1077 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:26:18.153823 1080 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:26:18.153872 1078 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:26:18.154956 1072 server_base.cc:1034] running on GCE node
I20260929 20:26:18.155663 1072 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:26:18.156821 1072 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:26:18.158030 1072 hybrid_clock.cc:648] HybridClock initialized: now 1790713578157970 us; error 68 us; skew 500 ppm
I20260929 20:26:18.158365 1072 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:26:18 dist-test-slave-zbnc krb5kdc[699](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713578, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.53.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:26:18.161800 1072 init.cc:377] Logged in from keytab as kudu/127.17.53.67@KRBTEST.COM (short username kudu)
I20260929 20:26:18.163509 1072 webserver.cc:460] Webserver started at http://127.17.53.67:41667/ using document root <none> and password file <none>
I20260929 20:26:18.164224 1072 fs_manager.cc:359] Metadata directory not provided
I20260929 20:26:18.164323 1072 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:26:18.164572 1072 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:26:18.167528 1072 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713339161492-17621-0/minicluster-data/ts-2/data/instance:
uuid: "445d9c75756f4730a2cb5bb98e8814e9"
format_stamp: "Formatted at 2026-09-29 20:26:18 on dist-test-slave-zbnc"
server_key: "730b50875bdd2ab9fe8dd6c354970414"
server_key_iv: "a91a7eb8fc8fda82009f81892050fd7d"
server_key_version: "encryptionkey@0"
I20260929 20:26:18.168395 1072 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713339161492-17621-0/minicluster-data/ts-2/wal/instance:
uuid: "445d9c75756f4730a2cb5bb98e8814e9"
format_stamp: "Formatted at 2026-09-29 20:26:18 on dist-test-slave-zbnc"
server_key: "730b50875bdd2ab9fe8dd6c354970414"
server_key_iv: "a91a7eb8fc8fda82009f81892050fd7d"
server_key_version: "encryptionkey@0"
I20260929 20:26:18.173390 1072 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260929 20:26:18.176954 1087 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:26:18.178196 1072 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 20:26:18.178370 1072 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713339161492-17621-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713339161492-17621-0/minicluster-data/ts-2/wal
uuid: "445d9c75756f4730a2cb5bb98e8814e9"
format_stamp: "Formatted at 2026-09-29 20:26:18 on dist-test-slave-zbnc"
server_key: "730b50875bdd2ab9fe8dd6c354970414"
server_key_iv: "a91a7eb8fc8fda82009f81892050fd7d"
server_key_version: "encryptionkey@0"
I20260929 20:26:18.178561 1072 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713339161492-17621-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713339161492-17621-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713339161492-17621-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:26:18.203678 1072 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:26:18.208145 1072 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:26:18.208397 1072 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:26:18.209290 1072 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:26:18.210529 1072 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:26:18.210601 1072 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:26:18.210716 1072 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:26:18.210760 1072 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:26:18.245431 1072 rpc_server.cc:307] RPC server started. Bound to: 127.17.53.67:45491
I20260929 20:26:18.245481 1199 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.53.67:45491 every 8 connection(s)
I20260929 20:26:18.246747 1072 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713339161492-17621-0/minicluster-data/ts-2/data/info.pb
I20260929 20:26:18.255097 17621 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu as pid 1072
I20260929 20:26:18.255318 17621 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713339161492-17621-0/minicluster-data/ts-2/wal/instance
I20260929 20:26:18.255889 17621 external_mini_cluster.cc:1428] Setting key 59217aad71f70093d4a7fce97ebd2e3e
I20260929 20:26:18.257577 933 heartbeater.cc:503] Master 127.17.53.126:39625 was elected leader, sending a full tablet report...
Sep 29 20:26:18 dist-test-slave-zbnc krb5kdc[699](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713578, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.53.67@KRBTEST.COM for kudu/127.17.53.126@KRBTEST.COM
I20260929 20:26:18.267773 935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:26:18.249504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.53.67:43171 (local address 127.17.53.126:39625)
0929 20:26:18.249705 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0929 20:26:18.249710 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:26:18.250950 (+ 1240us) server_negotiation.cc:407] Connection header received
0929 20:26:18.252856 (+ 1906us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:26:18.252860 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:26:18.252929 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:26:18.253017 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:26:18.254438 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:26:18.255315 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:26:18.257166 (+ 1851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:26:18.257860 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:26:18.257995 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:26:18.261750 (+ 3755us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:26:18.261756 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:26:18.261760 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:26:18.261803 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:26:18.264684 (+ 2881us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:26:18.265406 (+ 722us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:26:18.265412 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:26:18.265415 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:26:18.265546 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:26:18.265992 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:26:18.265999 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:26:18.266003 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:26:18.266276 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:26:18.266433 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0929 20:26:18.267200 (+ 767us) server_negotiation.cc:299] Negotiation successful
0929 20:26:18.267430 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260929 20:26:18.268735 1202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:26:18.250053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:39625 (local address 127.17.53.67:43171)
0929 20:26:18.250757 (+ 704us) negotiation.cc:107] Waiting for socket to connect
0929 20:26:18.250801 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 20:26:18.252593 (+ 1792us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:26:18.253238 (+ 645us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:26:18.253245 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:26:18.253736 (+ 491us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:26:18.254298 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:26:18.254315 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:26:18.255468 (+ 1153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:26:18.255477 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:26:18.257003 (+ 1526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:26:18.257021 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:26:18.258087 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:26:18.258095 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:26:18.258386 (+ 291us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:26:18.259210 (+ 824us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:26:18.259253 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:26:18.261528 (+ 2275us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:26:18.264829 (+ 3301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:26:18.264839 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:26:18.264855 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:26:18.265282 (+ 427us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:26:18.265659 (+ 377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:26:18.265664 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:26:18.265668 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:26:18.265876 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:26:18.266406 (+ 530us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:26:18.266414 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:26:18.266765 (+ 351us) client_negotiation.cc:769] Sending connection context
0929 20:26:18.267090 (+ 325us) client_negotiation.cc:240] Negotiation successful
0929 20:26:18.267638 (+ 548us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":485,"thread_start_us":284,"threads_started":1}
I20260929 20:26:18.270308 1200 heartbeater.cc:347] Connected to a master server at 127.17.53.126:39625
I20260929 20:26:18.270660 1200 heartbeater.cc:464] Registering TS with master...
I20260929 20:26:18.271605 1200 heartbeater.cc:511] Master 127.17.53.126:39625 requested a full tablet report, sending...
I20260929 20:26:18.273180 743 ts_manager.cc:194] Registered new tserver with Master: 445d9c75756f4730a2cb5bb98e8814e9 (127.17.53.67:45491)
I20260929 20:26:18.274036 743 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.53.67@KRBTEST.COM'} at 127.17.53.67:43171
I20260929 20:26:18.286151 17621 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 20:26:18.304683 935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:26:18.290692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49900 (local address 127.17.53.126:39625)
0929 20:26:18.290949 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0929 20:26:18.290958 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 20:26:18.291439 (+ 481us) server_negotiation.cc:407] Connection header received
0929 20:26:18.291678 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:26:18.291682 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:26:18.291752 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:26:18.291865 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:26:18.293453 (+ 1588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:26:18.294229 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:26:18.295892 (+ 1663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:26:18.296380 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:26:18.296540 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:26:18.299417 (+ 2877us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:26:18.299422 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:26:18.299425 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:26:18.299457 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:26:18.301839 (+ 2382us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:26:18.302623 (+ 784us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:26:18.302633 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:26:18.302635 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:26:18.302760 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:26:18.303214 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:26:18.303217 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:26:18.303220 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:26:18.303546 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:26:18.303804 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0929 20:26:18.304138 (+ 334us) server_negotiation.cc:299] Negotiation successful
0929 20:26:18.304352 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260929 20:26:18.308991 743 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49900:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 20:26:18.315901 743 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 20:26:18.340893 1217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:26:18.331302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50636 (local address 127.17.53.65:43287)
0929 20:26:18.332414 (+ 1112us) server_negotiation.cc:206] Beginning negotiation
0929 20:26:18.332423 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 20:26:18.333608 (+ 1185us) server_negotiation.cc:407] Connection header received
0929 20:26:18.333807 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:26:18.333813 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:26:18.333972 (+ 159us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:26:18.334145 (+ 173us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:26:18.334761 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:26:18.336393 (+ 1632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:26:18.338304 (+ 1911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:26:18.339665 (+ 1361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:26:18.339823 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:26:18.340069 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0929 20:26:18.340382 (+ 313us) server_negotiation.cc:299] Negotiation successful
0929 20:26:18.340542 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":930,"thread_start_us":776,"threads_started":1}
I20260929 20:26:18.340930 1216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:26:18.330631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.65:43287 (local address 127.0.0.1:50636)
0929 20:26:18.333500 (+ 2869us) negotiation.cc:107] Waiting for socket to connect
0929 20:26:18.333514 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 20:26:18.333643 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:26:18.334136 (+ 493us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:26:18.334140 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:26:18.334162 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:26:18.334547 (+ 385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:26:18.334560 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:26:18.336531 (+ 1971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:26:18.336537 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:26:18.338166 (+ 1629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:26:18.338178 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:26:18.339899 (+ 1721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:26:18.339906 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:26:18.340150 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:26:18.340208 (+ 58us) client_negotiation.cc:769] Sending connection context
0929 20:26:18.340338 (+ 130us) client_negotiation.cc:240] Negotiation successful
0929 20:26:18.340529 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2768,"mutex_wait_us":71,"thread_start_us":2633,"threads_started":1}
I20260929 20:26:18.341012 1213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:26:18.327741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47196 (local address 127.17.53.66:33821)
0929 20:26:18.331284 (+ 3543us) server_negotiation.cc:206] Beginning negotiation
0929 20:26:18.331293 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 20:26:18.331322 (+ 29us) server_negotiation.cc:407] Connection header received
0929 20:26:18.331421 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:26:18.331429 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:26:18.331736 (+ 307us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:26:18.332138 (+ 402us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:26:18.333276 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:26:18.334766 (+ 1490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:26:18.337742 (+ 2976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:26:18.339219 (+ 1477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:26:18.339350 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:26:18.339552 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0929 20:26:18.340505 (+ 953us) server_negotiation.cc:299] Negotiation successful
0929 20:26:18.340685 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3376,"thread_start_us":3232,"threads_started":1}
I20260929 20:26:18.341219 1215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:26:18.328708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48026 (local address 127.17.53.67:45491)
0929 20:26:18.331305 (+ 2597us) server_negotiation.cc:206] Beginning negotiation
0929 20:26:18.331314 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 20:26:18.331346 (+ 32us) server_negotiation.cc:407] Connection header received
0929 20:26:18.331444 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:26:18.331450 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:26:18.331718 (+ 268us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:26:18.332616 (+ 898us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:26:18.332893 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:26:18.334423 (+ 1530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:26:18.337733 (+ 3310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:26:18.339230 (+ 1497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:26:18.339350 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:26:18.339585 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0929 20:26:18.340698 (+ 1113us) server_negotiation.cc:299] Negotiation successful
0929 20:26:18.340867 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1528,"thread_start_us":1305,"threads_started":1}
I20260929 20:26:18.341737 1214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:26:18.327957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.67:45491 (local address 127.0.0.1:48026)
0929 20:26:18.330536 (+ 2579us) negotiation.cc:107] Waiting for socket to connect
0929 20:26:18.330555 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 20:26:18.330711 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:26:18.331963 (+ 1252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:26:18.331972 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:26:18.332024 (+ 52us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:26:18.332475 (+ 451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:26:18.332493 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:26:18.334576 (+ 2083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:26:18.334581 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:26:18.337543 (+ 2962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:26:18.337558 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:26:18.339769 (+ 2211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:26:18.339776 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:26:18.340015 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:26:18.340211 (+ 196us) client_negotiation.cc:769] Sending connection context
0929 20:26:18.340327 (+ 116us) client_negotiation.cc:240] Negotiation successful
0929 20:26:18.340531 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2405,"mutex_wait_us":306,"spinlock_wait_cycles":2432,"thread_start_us":2294,"threads_started":1}
I20260929 20:26:18.342031 1212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:26:18.327507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.66:33821 (local address 127.0.0.1:47196)
0929 20:26:18.329372 (+ 1865us) negotiation.cc:107] Waiting for socket to connect
0929 20:26:18.329431 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0929 20:26:18.329636 (+ 205us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:26:18.331975 (+ 2339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:26:18.331983 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:26:18.332040 (+ 57us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:26:18.333055 (+ 1015us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:26:18.333073 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:26:18.334941 (+ 1868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:26:18.334950 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:26:18.337546 (+ 2596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:26:18.337558 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:26:18.339718 (+ 2160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:26:18.339728 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:26:18.340013 (+ 285us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:26:18.340211 (+ 198us) client_negotiation.cc:769] Sending connection context
0929 20:26:18.340328 (+ 117us) client_negotiation.cc:240] Negotiation successful
0929 20:26:18.341299 (+ 971us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1525,"mutex_wait_us":970,"spinlock_wait_cycles":9344,"thread_start_us":1364,"threads_started":1}
I20260929 20:26:18.345328 1135 tablet_service.cc:1467] Processing CreateTablet for tablet 8851e4f6a6084f40b12849d5cf2732c0 (DEFAULT_TABLE table=test-table [id=3f5c94a78bd740da8fdd123f3916054d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:26:18.345503 1002 tablet_service.cc:1467] Processing CreateTablet for tablet 8851e4f6a6084f40b12849d5cf2732c0 (DEFAULT_TABLE table=test-table [id=3f5c94a78bd740da8fdd123f3916054d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:26:18.345388 868 tablet_service.cc:1467] Processing CreateTablet for tablet 8851e4f6a6084f40b12849d5cf2732c0 (DEFAULT_TABLE table=test-table [id=3f5c94a78bd740da8fdd123f3916054d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:26:18.346752 1002 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8851e4f6a6084f40b12849d5cf2732c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:26:18.346752 868 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8851e4f6a6084f40b12849d5cf2732c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:26:18.346787 1135 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8851e4f6a6084f40b12849d5cf2732c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:26:18.358824 1218 tablet_bootstrap.cc:492] T 8851e4f6a6084f40b12849d5cf2732c0 P 445d9c75756f4730a2cb5bb98e8814e9: Bootstrap starting.
I20260929 20:26:18.361359 1219 tablet_bootstrap.cc:492] T 8851e4f6a6084f40b12849d5cf2732c0 P 6e489a4674b840c4b61f170102698b7d: Bootstrap starting.
I20260929 20:26:18.362308 1220 tablet_bootstrap.cc:492] T 8851e4f6a6084f40b12849d5cf2732c0 P 3b59fcbacafc43b088c6cacc664f6ccf: Bootstrap starting.
I20260929 20:26:18.362870 1218 tablet_bootstrap.cc:654] T 8851e4f6a6084f40b12849d5cf2732c0 P 445d9c75756f4730a2cb5bb98e8814e9: Neither blocks nor log segments found. Creating new log.
I20260929 20:26:18.364243 1218 log.cc:824] T 8851e4f6a6084f40b12849d5cf2732c0 P 445d9c75756f4730a2cb5bb98e8814e9: Log is configured to *not* fsync() on all Append() calls
I20260929 20:26:18.365711 1220 tablet_bootstrap.cc:654] T 8851e4f6a6084f40b12849d5cf2732c0 P 3b59fcbacafc43b088c6cacc664f6ccf: Neither blocks nor log segments found. Creating new log.
I20260929 20:26:18.366043 1219 tablet_bootstrap.cc:654] T 8851e4f6a6084f40b12849d5cf2732c0 P 6e489a4674b840c4b61f170102698b7d: Neither blocks nor log segments found. Creating new log.
I20260929 20:26:18.366930 1220 log.cc:824] T 8851e4f6a6084f40b12849d5cf2732c0 P 3b59fcbacafc43b088c6cacc664f6ccf: Log is configured to *not* fsync() on all Append() calls
I20260929 20:26:18.367259 1219 log.cc:824] T 8851e4f6a6084f40b12849d5cf2732c0 P 6e489a4674b840c4b61f170102698b7d: Log is configured to *not* fsync() on all Append() calls
I20260929 20:26:18.368263 1218 tablet_bootstrap.cc:492] T 8851e4f6a6084f40b12849d5cf2732c0 P 445d9c75756f4730a2cb5bb98e8814e9: No bootstrap required, opened a new log
I20260929 20:26:18.368595 1218 ts_tablet_manager.cc:1397] T 8851e4f6a6084f40b12849d5cf2732c0 P 445d9c75756f4730a2cb5bb98e8814e9: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.004s
I20260929 20:26:18.369661 1219 tablet_bootstrap.cc:492] T 8851e4f6a6084f40b12849d5cf2732c0 P 6e489a4674b840c4b61f170102698b7d: No bootstrap required, opened a new log
I20260929 20:26:18.369911 1219 ts_tablet_manager.cc:1397] T 8851e4f6a6084f40b12849d5cf2732c0 P 6e489a4674b840c4b61f170102698b7d: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260929 20:26:18.376111 1220 tablet_bootstrap.cc:492] T 8851e4f6a6084f40b12849d5cf2732c0 P 3b59fcbacafc43b088c6cacc664f6ccf: No bootstrap required, opened a new log
I20260929 20:26:18.376391 1220 ts_tablet_manager.cc:1397] T 8851e4f6a6084f40b12849d5cf2732c0 P 3b59fcbacafc43b088c6cacc664f6ccf: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.008s
I20260929 20:26:18.378427 1219 raft_consensus.cc:348] T 8851e4f6a6084f40b12849d5cf2732c0 P 6e489a4674b840c4b61f170102698b7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b59fcbacafc43b088c6cacc664f6ccf" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 33821 } } peers { permanent_uuid: "445d9c75756f4730a2cb5bb98e8814e9" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 45491 } } peers { permanent_uuid: "6e489a4674b840c4b61f170102698b7d" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 43287 } }
I20260929 20:26:18.378939 1219 raft_consensus.cc:374] T 8851e4f6a6084f40b12849d5cf2732c0 P 6e489a4674b840c4b61f170102698b7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:26:18.379069 1219 raft_consensus.cc:729] T 8851e4f6a6084f40b12849d5cf2732c0 P 6e489a4674b840c4b61f170102698b7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e489a4674b840c4b61f170102698b7d, State: Initialized, Role: FOLLOWER
I20260929 20:26:18.379195 1218 raft_consensus.cc:348] T 8851e4f6a6084f40b12849d5cf2732c0 P 445d9c75756f4730a2cb5bb98e8814e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b59fcbacafc43b088c6cacc664f6ccf" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 33821 } } peers { permanent_uuid: "445d9c75756f4730a2cb5bb98e8814e9" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 45491 } } peers { permanent_uuid: "6e489a4674b840c4b61f170102698b7d" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 43287 } }
I20260929 20:26:18.379571 1218 raft_consensus.cc:374] T 8851e4f6a6084f40b12849d5cf2732c0 P 445d9c75756f4730a2cb5bb98e8814e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:26:18.379686 1218 raft_consensus.cc:729] T 8851e4f6a6084f40b12849d5cf2732c0 P 445d9c75756f4730a2cb5bb98e8814e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 445d9c75756f4730a2cb5bb98e8814e9, State: Initialized, Role: FOLLOWER
I20260929 20:26:18.379828 1219 consensus_queue.cc:260] T 8851e4f6a6084f40b12849d5cf2732c0 P 6e489a4674b840c4b61f170102698b7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b59fcbacafc43b088c6cacc664f6ccf" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 33821 } } peers { permanent_uuid: "445d9c75756f4730a2cb5bb98e8814e9" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 45491 } } peers { permanent_uuid: "6e489a4674b840c4b61f170102698b7d" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 43287 } }
I20260929 20:26:18.380293 1218 consensus_queue.cc:260] T 8851e4f6a6084f40b12849d5cf2732c0 P 445d9c75756f4730a2cb5bb98e8814e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b59fcbacafc43b088c6cacc664f6ccf" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 33821 } } peers { permanent_uuid: "445d9c75756f4730a2cb5bb98e8814e9" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 45491 } } peers { permanent_uuid: "6e489a4674b840c4b61f170102698b7d" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 43287 } }
I20260929 20:26:18.381659 1219 ts_tablet_manager.cc:1428] T 8851e4f6a6084f40b12849d5cf2732c0 P 6e489a4674b840c4b61f170102698b7d: Time spent starting tablet: real 0.012s user 0.012s sys 0.000s
I20260929 20:26:18.383484 1200 heartbeater.cc:503] Master 127.17.53.126:39625 was elected leader, sending a full tablet report...
I20260929 20:26:18.384145 1218 ts_tablet_manager.cc:1428] T 8851e4f6a6084f40b12849d5cf2732c0 P 445d9c75756f4730a2cb5bb98e8814e9: Time spent starting tablet: real 0.015s user 0.013s sys 0.003s
I20260929 20:26:18.386688 1220 raft_consensus.cc:348] T 8851e4f6a6084f40b12849d5cf2732c0 P 3b59fcbacafc43b088c6cacc664f6ccf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b59fcbacafc43b088c6cacc664f6ccf" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 33821 } } peers { permanent_uuid: "445d9c75756f4730a2cb5bb98e8814e9" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 45491 } } peers { permanent_uuid: "6e489a4674b840c4b61f170102698b7d" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 43287 } }
I20260929 20:26:18.387120 1220 raft_consensus.cc:374] T 8851e4f6a6084f40b12849d5cf2732c0 P 3b59fcbacafc43b088c6cacc664f6ccf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:26:18.387236 1220 raft_consensus.cc:729] T 8851e4f6a6084f40b12849d5cf2732c0 P 3b59fcbacafc43b088c6cacc664f6ccf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b59fcbacafc43b088c6cacc664f6ccf, State: Initialized, Role: FOLLOWER
I20260929 20:26:18.387867 1220 consensus_queue.cc:260] T 8851e4f6a6084f40b12849d5cf2732c0 P 3b59fcbacafc43b088c6cacc664f6ccf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b59fcbacafc43b088c6cacc664f6ccf" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 33821 } } peers { permanent_uuid: "445d9c75756f4730a2cb5bb98e8814e9" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 45491 } } peers { permanent_uuid: "6e489a4674b840c4b61f170102698b7d" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 43287 } }
I20260929 20:26:18.388985 1067 heartbeater.cc:503] Master 127.17.53.126:39625 was elected leader, sending a full tablet report...
I20260929 20:26:18.389492 1220 ts_tablet_manager.cc:1428] T 8851e4f6a6084f40b12849d5cf2732c0 P 3b59fcbacafc43b088c6cacc664f6ccf: Time spent starting tablet: real 0.013s user 0.003s sys 0.011s
I20260929 20:26:18.444231 1226 raft_consensus.cc:482] T 8851e4f6a6084f40b12849d5cf2732c0 P 3b59fcbacafc43b088c6cacc664f6ccf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 20:26:18.444499 1226 raft_consensus.cc:504] T 8851e4f6a6084f40b12849d5cf2732c0 P 3b59fcbacafc43b088c6cacc664f6ccf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b59fcbacafc43b088c6cacc664f6ccf" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 33821 } } peers { permanent_uuid: "445d9c75756f4730a2cb5bb98e8814e9" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 45491 } } peers { permanent_uuid: "6e489a4674b840c4b61f170102698b7d" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 43287 } }
I20260929 20:26:18.446118 1226 leader_election.cc:288] T 8851e4f6a6084f40b12849d5cf2732c0 P 3b59fcbacafc43b088c6cacc664f6ccf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 445d9c75756f4730a2cb5bb98e8814e9 (127.17.53.67:45491), 6e489a4674b840c4b61f170102698b7d (127.17.53.65:43287)
I20260929 20:26:18.453784 1215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:26:18.446613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.53.66:53807 (local address 127.17.53.67:45491)
0929 20:26:18.446805 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0929 20:26:18.446810 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:26:18.447251 (+ 441us) server_negotiation.cc:407] Connection header received
0929 20:26:18.447473 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:26:18.447479 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:26:18.447559 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:26:18.447700 (+ 141us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:26:18.448296 (+ 596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:26:18.449203 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:26:18.451809 (+ 2606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:26:18.452568 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:26:18.453014 (+ 446us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:26:18.453098 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0929 20:26:18.453408 (+ 310us) server_negotiation.cc:299] Negotiation successful
0929 20:26:18.453536 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260929 20:26:18.453871 1227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:26:18.446439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.67:45491 (local address 127.17.53.66:53807)
0929 20:26:18.447114 (+ 675us) negotiation.cc:107] Waiting for socket to connect
0929 20:26:18.447135 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 20:26:18.447312 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:26:18.447727 (+ 415us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:26:18.447731 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:26:18.447758 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:26:18.448146 (+ 388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:26:18.448162 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:26:18.449363 (+ 1201us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:26:18.449368 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:26:18.451648 (+ 2280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:26:18.451667 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:26:18.453062 (+ 1395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:26:18.453068 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:26:18.453238 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:26:18.453261 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 20:26:18.453368 (+ 107us) client_negotiation.cc:240] Negotiation successful
0929 20:26:18.453493 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":519,"thread_start_us":323,"threads_started":1}
I20260929 20:26:18.453857 1217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:26:18.447426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.53.66:42697 (local address 127.17.53.65:43287)
0929 20:26:18.447644 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0929 20:26:18.447652 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 20:26:18.447966 (+ 314us) server_negotiation.cc:407] Connection header received
0929 20:26:18.448166 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:26:18.448173 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:26:18.448235 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:26:18.448330 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:26:18.448896 (+ 566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:26:18.450194 (+ 1298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:26:18.452117 (+ 1923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:26:18.452922 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:26:18.453103 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:26:18.453179 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0929 20:26:18.453474 (+ 295us) server_negotiation.cc:299] Negotiation successful
0929 20:26:18.453600 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260929 20:26:18.454109 1228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:26:18.447247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.65:43287 (local address 127.17.53.66:42697)
0929 20:26:18.447870 (+ 623us) negotiation.cc:107] Waiting for socket to connect
0929 20:26:18.447883 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0929 20:26:18.448031 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:26:18.448344 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:26:18.448348 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:26:18.448371 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:26:18.448725 (+ 354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:26:18.448735 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:26:18.450342 (+ 1607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:26:18.450349 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:26:18.451965 (+ 1616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:26:18.451977 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:26:18.453141 (+ 1164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:26:18.453147 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:26:18.453317 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:26:18.453340 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 20:26:18.453439 (+ 99us) client_negotiation.cc:240] Negotiation successful
0929 20:26:18.453558 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":490,"thread_start_us":421,"threads_started":1}
I20260929 20:26:18.455134 1155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8851e4f6a6084f40b12849d5cf2732c0" candidate_uuid: "3b59fcbacafc43b088c6cacc664f6ccf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "445d9c75756f4730a2cb5bb98e8814e9" is_pre_election: true
I20260929 20:26:18.455122 888 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8851e4f6a6084f40b12849d5cf2732c0" candidate_uuid: "3b59fcbacafc43b088c6cacc664f6ccf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e489a4674b840c4b61f170102698b7d" is_pre_election: true
I20260929 20:26:18.455667 888 raft_consensus.cc:2454] T 8851e4f6a6084f40b12849d5cf2732c0 P 6e489a4674b840c4b61f170102698b7d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b59fcbacafc43b088c6cacc664f6ccf in term 0.
I20260929 20:26:18.455837 1155 raft_consensus.cc:2454] T 8851e4f6a6084f40b12849d5cf2732c0 P 445d9c75756f4730a2cb5bb98e8814e9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b59fcbacafc43b088c6cacc664f6ccf in term 0.
I20260929 20:26:18.456370 956 leader_election.cc:302] T 8851e4f6a6084f40b12849d5cf2732c0 P 3b59fcbacafc43b088c6cacc664f6ccf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b59fcbacafc43b088c6cacc664f6ccf, 6e489a4674b840c4b61f170102698b7d; no voters:
I20260929 20:26:18.456772 1226 raft_consensus.cc:2781] T 8851e4f6a6084f40b12849d5cf2732c0 P 3b59fcbacafc43b088c6cacc664f6ccf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 20:26:18.456916 1226 raft_consensus.cc:482] T 8851e4f6a6084f40b12849d5cf2732c0 P 3b59fcbacafc43b088c6cacc664f6ccf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 20:26:18.457099 1226 raft_consensus.cc:3037] T 8851e4f6a6084f40b12849d5cf2732c0 P 3b59fcbacafc43b088c6cacc664f6ccf [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:26:18.459609 1226 raft_consensus.cc:504] T 8851e4f6a6084f40b12849d5cf2732c0 P 3b59fcbacafc43b088c6cacc664f6ccf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b59fcbacafc43b088c6cacc664f6ccf" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 33821 } } peers { permanent_uuid: "445d9c75756f4730a2cb5bb98e8814e9" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 45491 } } peers { permanent_uuid: "6e489a4674b840c4b61f170102698b7d" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 43287 } }
I20260929 20:26:18.460145 1226 leader_election.cc:288] T 8851e4f6a6084f40b12849d5cf2732c0 P 3b59fcbacafc43b088c6cacc664f6ccf [CANDIDATE]: Term 1 election: Requested vote from peers 445d9c75756f4730a2cb5bb98e8814e9 (127.17.53.67:45491), 6e489a4674b840c4b61f170102698b7d (127.17.53.65:43287)
I20260929 20:26:18.460609 1155 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8851e4f6a6084f40b12849d5cf2732c0" candidate_uuid: "3b59fcbacafc43b088c6cacc664f6ccf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "445d9c75756f4730a2cb5bb98e8814e9"
I20260929 20:26:18.460646 888 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8851e4f6a6084f40b12849d5cf2732c0" candidate_uuid: "3b59fcbacafc43b088c6cacc664f6ccf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e489a4674b840c4b61f170102698b7d"
I20260929 20:26:18.460816 1155 raft_consensus.cc:3037] T 8851e4f6a6084f40b12849d5cf2732c0 P 445d9c75756f4730a2cb5bb98e8814e9 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:26:18.460940 888 raft_consensus.cc:3037] T 8851e4f6a6084f40b12849d5cf2732c0 P 6e489a4674b840c4b61f170102698b7d [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:26:18.463369 1155 raft_consensus.cc:2454] T 8851e4f6a6084f40b12849d5cf2732c0 P 445d9c75756f4730a2cb5bb98e8814e9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b59fcbacafc43b088c6cacc664f6ccf in term 1.
I20260929 20:26:18.463456 888 raft_consensus.cc:2454] T 8851e4f6a6084f40b12849d5cf2732c0 P 6e489a4674b840c4b61f170102698b7d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b59fcbacafc43b088c6cacc664f6ccf in term 1.
I20260929 20:26:18.463812 956 leader_election.cc:302] T 8851e4f6a6084f40b12849d5cf2732c0 P 3b59fcbacafc43b088c6cacc664f6ccf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b59fcbacafc43b088c6cacc664f6ccf, 445d9c75756f4730a2cb5bb98e8814e9; no voters:
I20260929 20:26:18.464115 1226 raft_consensus.cc:2781] T 8851e4f6a6084f40b12849d5cf2732c0 P 3b59fcbacafc43b088c6cacc664f6ccf [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:26:18.464556 1226 raft_consensus.cc:686] T 8851e4f6a6084f40b12849d5cf2732c0 P 3b59fcbacafc43b088c6cacc664f6ccf [term 1 LEADER]: Becoming Leader. State: Replica: 3b59fcbacafc43b088c6cacc664f6ccf, State: Running, Role: LEADER
I20260929 20:26:18.465117 1226 consensus_queue.cc:237] T 8851e4f6a6084f40b12849d5cf2732c0 P 3b59fcbacafc43b088c6cacc664f6ccf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b59fcbacafc43b088c6cacc664f6ccf" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 33821 } } peers { permanent_uuid: "445d9c75756f4730a2cb5bb98e8814e9" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 45491 } } peers { permanent_uuid: "6e489a4674b840c4b61f170102698b7d" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 43287 } }
I20260929 20:26:18.469720 743 catalog_manager.cc:5495] T 8851e4f6a6084f40b12849d5cf2732c0 P 3b59fcbacafc43b088c6cacc664f6ccf reported cstate change: term changed from 0 to 1, leader changed from <none> to 3b59fcbacafc43b088c6cacc664f6ccf (127.17.53.66). New cstate: current_term: 1 leader_uuid: "3b59fcbacafc43b088c6cacc664f6ccf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b59fcbacafc43b088c6cacc664f6ccf" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 33821 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "445d9c75756f4730a2cb5bb98e8814e9" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 45491 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6e489a4674b840c4b61f170102698b7d" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 43287 } health_report { overall_health: UNKNOWN } } }
W20260929 20:26:18.474742 934 tablet.cc:2369] T 8851e4f6a6084f40b12849d5cf2732c0 P 6e489a4674b840c4b61f170102698b7d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 20:26:18.486156 1068 tablet.cc:2369] T 8851e4f6a6084f40b12849d5cf2732c0 P 3b59fcbacafc43b088c6cacc664f6ccf: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 20:26:18.492130 1213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:26:18.486798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47200 (local address 127.17.53.66:33821)
0929 20:26:18.487045 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0929 20:26:18.487052 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:26:18.487072 (+ 20us) server_negotiation.cc:407] Connection header received
0929 20:26:18.487169 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:26:18.487173 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:26:18.487254 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:26:18.487406 (+ 152us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 20:26:18.487814 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:26:18.488516 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:26:18.490068 (+ 1552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:26:18.490473 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:26:18.490574 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:26:18.491084 (+ 510us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 20:26:18.491575 (+ 491us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 20:26:18.491708 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0929 20:26:18.491767 (+ 59us) server_negotiation.cc:299] Negotiation successful
0929 20:26:18.491832 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
W20260929 20:26:18.498394 1201 tablet.cc:2369] T 8851e4f6a6084f40b12849d5cf2732c0 P 445d9c75756f4730a2cb5bb98e8814e9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 20:26:18.500279 1155 raft_consensus.cc:1261] T 8851e4f6a6084f40b12849d5cf2732c0 P 445d9c75756f4730a2cb5bb98e8814e9 [term 1 FOLLOWER]: Refusing update from remote peer 3b59fcbacafc43b088c6cacc664f6ccf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:26:18.500280 888 raft_consensus.cc:1261] T 8851e4f6a6084f40b12849d5cf2732c0 P 6e489a4674b840c4b61f170102698b7d [term 1 FOLLOWER]: Refusing update from remote peer 3b59fcbacafc43b088c6cacc664f6ccf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:26:18.501245 1226 consensus_queue.cc:1035] T 8851e4f6a6084f40b12849d5cf2732c0 P 3b59fcbacafc43b088c6cacc664f6ccf [LEADER]: Connected to new peer: Peer: permanent_uuid: "445d9c75756f4730a2cb5bb98e8814e9" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 45491 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:26:18.501562 1229 consensus_queue.cc:1035] T 8851e4f6a6084f40b12849d5cf2732c0 P 3b59fcbacafc43b088c6cacc664f6ccf [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e489a4674b840c4b61f170102698b7d" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 43287 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:26:18.521883 1233 mvcc.cc:205] Tried to move back new op lower bound from 7334762817526325248 to 7334762817397252096. Current Snapshot: MvccSnapshot[applied={T|T < 7334762817526325248}]
I20260929 20:26:18.528124 743 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49900:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 20:26:18.528412 743 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49900:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 20:26:18.532105 743 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e5f2bfb42723428996a3b2e3bb46f683: Sending DeleteTablet for 3 replicas of tablet 8851e4f6a6084f40b12849d5cf2732c0
I20260929 20:26:18.533224 1002 tablet_service.cc:1514] Processing DeleteTablet for tablet 8851e4f6a6084f40b12849d5cf2732c0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:26:18 UTC) from {username='kudu', principal='kudu/127.17.53.126@KRBTEST.COM'} at 127.0.0.1:47196
I20260929 20:26:18.533651 17621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu pid 805
I20260929 20:26:18.533711 1135 tablet_service.cc:1514] Processing DeleteTablet for tablet 8851e4f6a6084f40b12849d5cf2732c0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:26:18 UTC) from {username='kudu', principal='kudu/127.17.53.126@KRBTEST.COM'} at 127.0.0.1:48026
I20260929 20:26:18.534416 868 tablet_service.cc:1514] Processing DeleteTablet for tablet 8851e4f6a6084f40b12849d5cf2732c0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:26:18 UTC) from {username='kudu', principal='kudu/127.17.53.126@KRBTEST.COM'} at 127.0.0.1:50636
I20260929 20:26:18.534688 1243 tablet_replica.cc:330] T 8851e4f6a6084f40b12849d5cf2732c0 P 3b59fcbacafc43b088c6cacc664f6ccf: stopping tablet replica
I20260929 20:26:18.535148 1243 raft_consensus.cc:2229] T 8851e4f6a6084f40b12849d5cf2732c0 P 3b59fcbacafc43b088c6cacc664f6ccf [term 1 LEADER]: Raft consensus shutting down.
I20260929 20:26:18.535933 1243 raft_consensus.cc:2258] T 8851e4f6a6084f40b12849d5cf2732c0 P 3b59fcbacafc43b088c6cacc664f6ccf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:26:18.536849 1246 tablet_replica.cc:330] T 8851e4f6a6084f40b12849d5cf2732c0 P 6e489a4674b840c4b61f170102698b7d: stopping tablet replica
I20260929 20:26:18.537364 1246 raft_consensus.cc:2229] T 8851e4f6a6084f40b12849d5cf2732c0 P 6e489a4674b840c4b61f170102698b7d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 20:26:18.537955 1246 raft_consensus.cc:2258] T 8851e4f6a6084f40b12849d5cf2732c0 P 6e489a4674b840c4b61f170102698b7d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:26:18.541024 1246 ts_tablet_manager.cc:1905] T 8851e4f6a6084f40b12849d5cf2732c0 P 6e489a4674b840c4b61f170102698b7d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:26:18.541147 1243 ts_tablet_manager.cc:1905] T 8851e4f6a6084f40b12849d5cf2732c0 P 3b59fcbacafc43b088c6cacc664f6ccf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:26:18.549023 1246 ts_tablet_manager.cc:1918] T 8851e4f6a6084f40b12849d5cf2732c0 P 6e489a4674b840c4b61f170102698b7d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 20:26:18.549208 1246 log.cc:1196] T 8851e4f6a6084f40b12849d5cf2732c0 P 6e489a4674b840c4b61f170102698b7d: Deleting WAL directory at /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713339161492-17621-0/minicluster-data/ts-0/wal/wals/8851e4f6a6084f40b12849d5cf2732c0
I20260929 20:26:18.549911 1246 ts_tablet_manager.cc:1939] T 8851e4f6a6084f40b12849d5cf2732c0 P 6e489a4674b840c4b61f170102698b7d: Deleting consensus metadata
Sep 29 20:26:18 dist-test-slave-zbnc krb5kdc[699](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713576, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.53.65@KRBTEST.COM
I20260929 20:26:18.552002 729 catalog_manager.cc:4841] TS 6e489a4674b840c4b61f170102698b7d (127.17.53.65:43287): tablet 8851e4f6a6084f40b12849d5cf2732c0 (table test-table [id=3f5c94a78bd740da8fdd123f3916054d]) successfully deleted
I20260929 20:26:18.552727 1243 ts_tablet_manager.cc:1918] T 8851e4f6a6084f40b12849d5cf2732c0 P 3b59fcbacafc43b088c6cacc664f6ccf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 20:26:18.552927 1243 log.cc:1196] T 8851e4f6a6084f40b12849d5cf2732c0 P 3b59fcbacafc43b088c6cacc664f6ccf: Deleting WAL directory at /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713339161492-17621-0/minicluster-data/ts-1/wal/wals/8851e4f6a6084f40b12849d5cf2732c0
I20260929 20:26:18.553486 1243 ts_tablet_manager.cc:1939] T 8851e4f6a6084f40b12849d5cf2732c0 P 3b59fcbacafc43b088c6cacc664f6ccf: Deleting consensus metadata
I20260929 20:26:18.555455 1244 tablet_replica.cc:330] T 8851e4f6a6084f40b12849d5cf2732c0 P 445d9c75756f4730a2cb5bb98e8814e9: stopping tablet replica
I20260929 20:26:18.555948 1244 raft_consensus.cc:2229] T 8851e4f6a6084f40b12849d5cf2732c0 P 445d9c75756f4730a2cb5bb98e8814e9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 20:26:18.556391 728 catalog_manager.cc:4841] TS 3b59fcbacafc43b088c6cacc664f6ccf (127.17.53.66:33821): tablet 8851e4f6a6084f40b12849d5cf2732c0 (table test-table [id=3f5c94a78bd740da8fdd123f3916054d]) successfully deleted
I20260929 20:26:18.556525 1244 raft_consensus.cc:2258] T 8851e4f6a6084f40b12849d5cf2732c0 P 445d9c75756f4730a2cb5bb98e8814e9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:26:18.558609 1244 ts_tablet_manager.cc:1905] T 8851e4f6a6084f40b12849d5cf2732c0 P 445d9c75756f4730a2cb5bb98e8814e9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:26:18.560002 1217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:26:18.534999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50650 (local address 127.17.53.65:43287)
0929 20:26:18.535181 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0929 20:26:18.535189 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 20:26:18.538884 (+ 3695us) server_negotiation.cc:407] Connection header received
0929 20:26:18.539109 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:26:18.539115 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:26:18.539209 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:26:18.539309 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:26:18.541695 (+ 2386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:26:18.542893 (+ 1198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:26:18.545004 (+ 2111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:26:18.545674 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:26:18.545779 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:26:18.551842 (+ 6063us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:26:18.551855 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:26:18.551873 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:26:18.551920 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:26:18.556683 (+ 4763us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:26:18.557652 (+ 969us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:26:18.557661 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:26:18.557665 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:26:18.557826 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:26:18.558277 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:26:18.558282 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:26:18.558285 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:26:18.558734 (+ 449us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:26:18.559066 (+ 332us) server_negotiation.cc:1035] Waiting for connection context
0929 20:26:18.559410 (+ 344us) server_negotiation.cc:299] Negotiation successful
0929 20:26:18.559628 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 20:26:18.560699 928 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:26:18.564704 1244 ts_tablet_manager.cc:1918] T 8851e4f6a6084f40b12849d5cf2732c0 P 445d9c75756f4730a2cb5bb98e8814e9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 20:26:18.564868 1244 log.cc:1196] T 8851e4f6a6084f40b12849d5cf2732c0 P 445d9c75756f4730a2cb5bb98e8814e9: Deleting WAL directory at /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713339161492-17621-0/minicluster-data/ts-2/wal/wals/8851e4f6a6084f40b12849d5cf2732c0
I20260929 20:26:18.565304 1244 ts_tablet_manager.cc:1939] T 8851e4f6a6084f40b12849d5cf2732c0 P 445d9c75756f4730a2cb5bb98e8814e9: Deleting consensus metadata
I20260929 20:26:18.566736 729 catalog_manager.cc:4841] TS 445d9c75756f4730a2cb5bb98e8814e9 (127.17.53.67:45491): tablet 8851e4f6a6084f40b12849d5cf2732c0 (table test-table [id=3f5c94a78bd740da8fdd123f3916054d]) successfully deleted
I20260929 20:26:18.657670 17621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu with pid 805
I20260929 20:26:18.676813 17621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu pid 939
Sep 29 20:26:18 dist-test-slave-zbnc krb5kdc[699](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713576, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.53.66@KRBTEST.COM
I20260929 20:26:18.696149 1213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:26:18.677868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47208 (local address 127.17.53.66:33821)
0929 20:26:18.678077 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0929 20:26:18.678084 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:26:18.678104 (+ 20us) server_negotiation.cc:407] Connection header received
0929 20:26:18.678170 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:26:18.678176 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:26:18.678290 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:26:18.678400 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:26:18.680068 (+ 1668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:26:18.681282 (+ 1214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:26:18.682906 (+ 1624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:26:18.683597 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:26:18.683728 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:26:18.688580 (+ 4852us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:26:18.688597 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:26:18.688617 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:26:18.688677 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:26:18.692808 (+ 4131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:26:18.693607 (+ 799us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:26:18.693617 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:26:18.693621 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:26:18.693805 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:26:18.694291 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:26:18.694296 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:26:18.694299 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:26:18.694765 (+ 466us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:26:18.695151 (+ 386us) server_negotiation.cc:1035] Waiting for connection context
0929 20:26:18.695488 (+ 337us) server_negotiation.cc:299] Negotiation successful
0929 20:26:18.695758 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260929 20:26:18.696964 1062 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:26:18.810997 17621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu with pid 939
I20260929 20:26:18.829859 17621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu pid 1072
Sep 29 20:26:18 dist-test-slave-zbnc krb5kdc[699](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713576, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.53.67@KRBTEST.COM
I20260929 20:26:18.846184 1215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:26:18.830577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48028 (local address 127.17.53.67:45491)
0929 20:26:18.830750 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0929 20:26:18.830756 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:26:18.830826 (+ 70us) server_negotiation.cc:407] Connection header received
0929 20:26:18.831062 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:26:18.831066 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:26:18.831129 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:26:18.831253 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:26:18.832649 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:26:18.833401 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:26:18.834764 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:26:18.835322 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:26:18.835461 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:26:18.840278 (+ 4817us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:26:18.840296 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:26:18.840313 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:26:18.840355 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:26:18.843333 (+ 2978us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:26:18.844031 (+ 698us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:26:18.844039 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:26:18.844041 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:26:18.844216 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:26:18.844580 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:26:18.844584 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:26:18.844587 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:26:18.844992 (+ 405us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:26:18.845249 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0929 20:26:18.845590 (+ 341us) server_negotiation.cc:299] Negotiation successful
0929 20:26:18.845851 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 20:26:18.846930 1195 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:26:18.936770 17621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu with pid 1072
I20260929 20:26:18.955850 17621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu pid 712
I20260929 20:26:18.957005 774 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:26:19.071557 17621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu with pid 712
I20260929 20:26:39.093518 17621 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
/tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713339161492-17621-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713339161492-17621-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713339161492-17621-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713339161492-17621-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.53.126:39625
--webserver_interface=127.17.53.126
--webserver_port=42051
--keytab_file=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.126.keytab
--principal=kudu/127.17.53.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.53.84:36709
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.53.126:39625
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713339161492-17621-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713339161492-17621-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713339161492-17621-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:26:39.430442 1251 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:26:39.430758 1251 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:26:39.430905 1251 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:26:39.438381 1251 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260929 20:26:39.438496 1251 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260929 20:26:39.438536 1251 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260929 20:26:39.438581 1251 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:26:39.438637 1251 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:26:39.438702 1251 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:26:39.438747 1251 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:26:39.438795 1251 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:26:39.449082 1251 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.53.84:36709
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713339161492-17621-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713339161492-17621-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.17.53.126:39625
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.126.keytab
--principal=kudu/127.17.53.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.53.126:39625
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713339161492-17621-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.53.126
--webserver_port=42051
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713339161492-17621-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:26:39.450405 1251 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:26:39.451810 1251 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:26:39.475785 1257 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:26:39.475832 1260 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:26:39.475805 1258 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:26:39.476809 1251 server_base.cc:1034] running on GCE node
I20260929 20:26:39.477735 1251 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:26:39.479434 1251 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:26:39.480690 1251 hybrid_clock.cc:648] HybridClock initialized: now 1790713599480613 us; error 39 us; skew 500 ppm
I20260929 20:26:39.481166 1251 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:26:39 dist-test-slave-zbnc krb5kdc[699](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713599, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.53.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:26:39.488523 1251 init.cc:377] Logged in from keytab as kudu/127.17.53.126@KRBTEST.COM (short username kudu)
I20260929 20:26:39.490486 1251 webserver.cc:460] Webserver started at http://127.17.53.126:42051/ using document root <none> and password file <none>
I20260929 20:26:39.491386 1251 fs_manager.cc:359] Metadata directory not provided
I20260929 20:26:39.491539 1251 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:26:39.497733 1251 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.003s
I20260929 20:26:39.500793 1267 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:26:39.502204 1251 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 20:26:39.502399 1251 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713339161492-17621-0/minicluster-data/master-0/data,/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713339161492-17621-0/minicluster-data/master-0/wal
uuid: "e5f2bfb42723428996a3b2e3bb46f683"
format_stamp: "Formatted at 2026-09-29 20:26:16 on dist-test-slave-zbnc"
server_key: "6f9bd922439cf9858a42a8d6ec9ce281"
server_key_iv: "dcf1d40a474d3e29ab4ba8f68c56b511"
server_key_version: "encryptionkey@0"
I20260929 20:26:39.503623 1251 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713339161492-17621-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713339161492-17621-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713339161492-17621-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:26:39.519579 1251 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:26:39.524394 1251 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:26:39.524735 1251 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:26:39.544602 1251 rpc_server.cc:307] RPC server started. Bound to: 127.17.53.126:39625
I20260929 20:26:39.544641 1318 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.53.126:39625 every 8 connection(s)
I20260929 20:26:39.546234 1251 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713339161492-17621-0/minicluster-data/master-0/data/info.pb
I20260929 20:26:39.548580 17621 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu as pid 1251
I20260929 20:26:39.549678 17621 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
/tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713339161492-17621-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713339161492-17621-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713339161492-17621-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713339161492-17621-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.53.65:43287
--local_ip_for_outbound_sockets=127.17.53.65
--tserver_master_addrs=127.17.53.126:39625
--webserver_port=33787
--webserver_interface=127.17.53.65
--keytab_file=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.65.keytab
--principal=kudu/127.17.53.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.53.84:36709
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713339161492-17621-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713339161492-17621-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713339161492-17621-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 20:26:39.557601 1319 sys_catalog.cc:263] Verifying existing consensus state
I20260929 20:26:39.562067 1319 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e5f2bfb42723428996a3b2e3bb46f683: Bootstrap starting.
I20260929 20:26:39.584075 1319 log.cc:824] T 00000000000000000000000000000000 P e5f2bfb42723428996a3b2e3bb46f683: Log is configured to *not* fsync() on all Append() calls
I20260929 20:26:39.597731 1319 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e5f2bfb42723428996a3b2e3bb46f683: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 20:26:39.598361 1319 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e5f2bfb42723428996a3b2e3bb46f683: Bootstrap complete.
I20260929 20:26:39.609767 1319 raft_consensus.cc:348] T 00000000000000000000000000000000 P e5f2bfb42723428996a3b2e3bb46f683 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5f2bfb42723428996a3b2e3bb46f683" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 39625 } }
I20260929 20:26:39.610728 1319 raft_consensus.cc:729] T 00000000000000000000000000000000 P e5f2bfb42723428996a3b2e3bb46f683 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5f2bfb42723428996a3b2e3bb46f683, State: Initialized, Role: FOLLOWER
I20260929 20:26:39.611421 1319 consensus_queue.cc:260] T 00000000000000000000000000000000 P e5f2bfb42723428996a3b2e3bb46f683 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5f2bfb42723428996a3b2e3bb46f683" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 39625 } }
I20260929 20:26:39.611693 1319 raft_consensus.cc:388] T 00000000000000000000000000000000 P e5f2bfb42723428996a3b2e3bb46f683 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 20:26:39.611832 1319 raft_consensus.cc:482] T 00000000000000000000000000000000 P e5f2bfb42723428996a3b2e3bb46f683 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 20:26:39.612047 1319 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e5f2bfb42723428996a3b2e3bb46f683 [term 1 FOLLOWER]: Advancing to term 2
I20260929 20:26:39.616357 1319 raft_consensus.cc:504] T 00000000000000000000000000000000 P e5f2bfb42723428996a3b2e3bb46f683 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5f2bfb42723428996a3b2e3bb46f683" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 39625 } }
I20260929 20:26:39.616991 1319 leader_election.cc:302] T 00000000000000000000000000000000 P e5f2bfb42723428996a3b2e3bb46f683 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e5f2bfb42723428996a3b2e3bb46f683; no voters:
I20260929 20:26:39.617707 1319 leader_election.cc:288] T 00000000000000000000000000000000 P e5f2bfb42723428996a3b2e3bb46f683 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260929 20:26:39.617872 1323 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e5f2bfb42723428996a3b2e3bb46f683 [term 2 FOLLOWER]: Leader election won for term 2
I20260929 20:26:39.619163 1323 raft_consensus.cc:686] T 00000000000000000000000000000000 P e5f2bfb42723428996a3b2e3bb46f683 [term 2 LEADER]: Becoming Leader. State: Replica: e5f2bfb42723428996a3b2e3bb46f683, State: Running, Role: LEADER
I20260929 20:26:39.619835 1323 consensus_queue.cc:237] T 00000000000000000000000000000000 P e5f2bfb42723428996a3b2e3bb46f683 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5f2bfb42723428996a3b2e3bb46f683" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 39625 } }
I20260929 20:26:39.620272 1319 sys_catalog.cc:564] T 00000000000000000000000000000000 P e5f2bfb42723428996a3b2e3bb46f683 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:26:39.622442 1325 sys_catalog.cc:455] T 00000000000000000000000000000000 P e5f2bfb42723428996a3b2e3bb46f683 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e5f2bfb42723428996a3b2e3bb46f683. Latest consensus state: current_term: 2 leader_uuid: "e5f2bfb42723428996a3b2e3bb46f683" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5f2bfb42723428996a3b2e3bb46f683" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 39625 } } }
I20260929 20:26:39.622900 1325 sys_catalog.cc:458] T 00000000000000000000000000000000 P e5f2bfb42723428996a3b2e3bb46f683 [sys.catalog]: This master's current role is: LEADER
I20260929 20:26:39.622681 1324 sys_catalog.cc:455] T 00000000000000000000000000000000 P e5f2bfb42723428996a3b2e3bb46f683 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "e5f2bfb42723428996a3b2e3bb46f683" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5f2bfb42723428996a3b2e3bb46f683" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 39625 } } }
I20260929 20:26:39.623198 1324 sys_catalog.cc:458] T 00000000000000000000000000000000 P e5f2bfb42723428996a3b2e3bb46f683 [sys.catalog]: This master's current role is: LEADER
I20260929 20:26:39.635880 1330 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:26:39.644582 1330 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:26:39.645956 1330 catalog_manager.cc:1259] Loaded cluster ID: 440651dfe5e2435689788eaff7deaa18
I20260929 20:26:39.646066 1330 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:26:39.650030 1330 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:26:39.651713 1330 catalog_manager.cc:5879] T 00000000000000000000000000000000 P e5f2bfb42723428996a3b2e3bb46f683: Loaded TSK: 0
I20260929 20:26:39.656540 1340 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260929 20:26:39.687222 1330 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e5f2bfb42723428996a3b2e3bb46f683: Generated new TSK 1
I20260929 20:26:39.687645 1330 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260929 20:26:39.912863 1321 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:26:39.913219 1321 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:26:39.913383 1321 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:26:39.921168 1321 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:26:39.921330 1321 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:26:39.921505 1321 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.53.65
I20260929 20:26:39.931888 1321 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.53.84:36709
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713339161492-17621-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713339161492-17621-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.65.keytab
--principal=kudu/127.17.53.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.53.65:43287
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713339161492-17621-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.17.53.65
--webserver_port=33787
--tserver_master_addrs=127.17.53.126:39625
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.53.65
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713339161492-17621-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:26:39.933341 1321 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:26:39.934728 1321 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:26:39.948098 1346 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:26:39.948329 1347 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:26:39.948508 1349 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:26:39.949168 1321 server_base.cc:1034] running on GCE node
I20260929 20:26:39.949800 1321 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:26:39.950786 1321 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:26:39.952039 1321 hybrid_clock.cc:648] HybridClock initialized: now 1790713599951977 us; error 66 us; skew 500 ppm
I20260929 20:26:39.952380 1321 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:26:39 dist-test-slave-zbnc krb5kdc[699](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713599, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.53.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:26:39.955749 1321 init.cc:377] Logged in from keytab as kudu/127.17.53.65@KRBTEST.COM (short username kudu)
I20260929 20:26:39.957540 1321 webserver.cc:460] Webserver started at http://127.17.53.65:33787/ using document root <none> and password file <none>
I20260929 20:26:39.958262 1321 fs_manager.cc:359] Metadata directory not provided
I20260929 20:26:39.958408 1321 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:26:39.964612 1321 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 20:26:39.967782 1357 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:26:39.969306 1321 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 20:26:39.969481 1321 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713339161492-17621-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713339161492-17621-0/minicluster-data/ts-0/wal
uuid: "6e489a4674b840c4b61f170102698b7d"
format_stamp: "Formatted at 2026-09-29 20:26:17 on dist-test-slave-zbnc"
server_key: "5684d05b4cfcfc1fb7d6fa54ad8e7c7b"
server_key_iv: "c257d9544ebce464fb7d5810b92342c8"
server_key_version: "encryptionkey@0"
I20260929 20:26:39.970476 1321 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713339161492-17621-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713339161492-17621-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713339161492-17621-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:26:40.000376 1321 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:26:40.005089 1321 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:26:40.005383 1321 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:26:40.006551 1321 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:26:40.008344 1321 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:26:40.008479 1321 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:26:40.008610 1321 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:26:40.008692 1321 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:26:40.048470 1321 rpc_server.cc:307] RPC server started. Bound to: 127.17.53.65:43287
I20260929 20:26:40.048527 1469 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.53.65:43287 every 8 connection(s)
I20260929 20:26:40.049906 1321 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713339161492-17621-0/minicluster-data/ts-0/data/info.pb
I20260929 20:26:40.058465 17621 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu as pid 1321
I20260929 20:26:40.059587 17621 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
/tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713339161492-17621-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713339161492-17621-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713339161492-17621-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713339161492-17621-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.53.66:33821
--local_ip_for_outbound_sockets=127.17.53.66
--tserver_master_addrs=127.17.53.126:39625
--webserver_port=37051
--webserver_interface=127.17.53.66
--keytab_file=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.66.keytab
--principal=kudu/127.17.53.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.53.84:36709
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713339161492-17621-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713339161492-17621-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713339161492-17621-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 20:26:40 dist-test-slave-zbnc krb5kdc[699](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713599, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.53.65@KRBTEST.COM for kudu/127.17.53.126@KRBTEST.COM
I20260929 20:26:40.084044 1472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:26:40.053724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.53.65:32997 (local address 127.17.53.126:39625)
0929 20:26:40.054675 (+ 951us) server_negotiation.cc:206] Beginning negotiation
0929 20:26:40.054695 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0929 20:26:40.055156 (+ 461us) server_negotiation.cc:407] Connection header received
0929 20:26:40.058404 (+ 3248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:26:40.058426 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:26:40.058948 (+ 522us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:26:40.059364 (+ 416us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:26:40.061909 (+ 2545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:26:40.063998 (+ 2089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:26:40.067773 (+ 3775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:26:40.068940 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:26:40.069155 (+ 215us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:26:40.074154 (+ 4999us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:26:40.074217 (+ 63us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:26:40.074244 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:26:40.074302 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:26:40.078683 (+ 4381us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:26:40.079766 (+ 1083us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:26:40.079778 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:26:40.079789 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:26:40.080022 (+ 233us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:26:40.080454 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:26:40.080469 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:26:40.080472 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:26:40.081253 (+ 781us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:26:40.081448 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0929 20:26:40.082292 (+ 844us) server_negotiation.cc:299] Negotiation successful
0929 20:26:40.082666 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":455,"thread_start_us":298,"threads_started":1}
I20260929 20:26:40.084048 1473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:26:40.053725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:39625 (local address 127.17.53.65:32997)
0929 20:26:40.054844 (+ 1119us) negotiation.cc:107] Waiting for socket to connect
0929 20:26:40.054913 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0929 20:26:40.057605 (+ 2692us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:26:40.059595 (+ 1990us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:26:40.059608 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:26:40.060579 (+ 971us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:26:40.061618 (+ 1039us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:26:40.061651 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:26:40.064869 (+ 3218us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:26:40.064878 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:26:40.067551 (+ 2673us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:26:40.067572 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:26:40.069580 (+ 2008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:26:40.069590 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:26:40.069924 (+ 334us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:26:40.070656 (+ 732us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:26:40.070679 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:26:40.073813 (+ 3134us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:26:40.078987 (+ 5174us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:26:40.079000 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:26:40.079021 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:26:40.079620 (+ 599us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:26:40.080147 (+ 527us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:26:40.080150 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:26:40.080152 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:26:40.080328 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:26:40.081414 (+ 1086us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:26:40.081423 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:26:40.081772 (+ 349us) client_negotiation.cc:769] Sending connection context
0929 20:26:40.082065 (+ 293us) client_negotiation.cc:240] Negotiation successful
0929 20:26:40.082641 (+ 576us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":834,"thread_start_us":638,"threads_started":1}
I20260929 20:26:40.088182 1470 heartbeater.cc:347] Connected to a master server at 127.17.53.126:39625
I20260929 20:26:40.088486 1470 heartbeater.cc:464] Registering TS with master...
I20260929 20:26:40.089318 1470 heartbeater.cc:511] Master 127.17.53.126:39625 requested a full tablet report, sending...
I20260929 20:26:40.091822 1284 ts_manager.cc:194] Registered new tserver with Master: 6e489a4674b840c4b61f170102698b7d (127.17.53.65:43287)
I20260929 20:26:40.093310 1284 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.53.65@KRBTEST.COM'} at 127.17.53.65:32997
W20260929 20:26:40.403880 1474 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:26:40.404208 1474 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:26:40.404386 1474 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:26:40.411842 1474 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:26:40.412014 1474 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:26:40.412200 1474 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.53.66
I20260929 20:26:40.423056 1474 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.53.84:36709
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713339161492-17621-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713339161492-17621-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.66.keytab
--principal=kudu/127.17.53.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.53.66:33821
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713339161492-17621-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.17.53.66
--webserver_port=37051
--tserver_master_addrs=127.17.53.126:39625
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.53.66
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713339161492-17621-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:26:40.424532 1474 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:26:40.425890 1474 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:26:40.440078 1482 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:26:40.440143 1479 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:26:40.440531 1480 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:26:40.441174 1474 server_base.cc:1034] running on GCE node
I20260929 20:26:40.441767 1474 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:26:40.442833 1474 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:26:40.444175 1474 hybrid_clock.cc:648] HybridClock initialized: now 1790713600444102 us; error 38 us; skew 500 ppm
I20260929 20:26:40.444550 1474 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:26:40 dist-test-slave-zbnc krb5kdc[699](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713600, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.53.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:26:40.448022 1474 init.cc:377] Logged in from keytab as kudu/127.17.53.66@KRBTEST.COM (short username kudu)
I20260929 20:26:40.449617 1474 webserver.cc:460] Webserver started at http://127.17.53.66:37051/ using document root <none> and password file <none>
I20260929 20:26:40.450352 1474 fs_manager.cc:359] Metadata directory not provided
I20260929 20:26:40.450500 1474 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:26:40.456277 1474 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.002s
I20260929 20:26:40.459311 1489 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:26:40.460765 1474 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260929 20:26:40.460970 1474 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713339161492-17621-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713339161492-17621-0/minicluster-data/ts-1/wal
uuid: "3b59fcbacafc43b088c6cacc664f6ccf"
format_stamp: "Formatted at 2026-09-29 20:26:17 on dist-test-slave-zbnc"
server_key: "79ab5d60f09934e4ae2ad1cda3c47a82"
server_key_iv: "a2995c358f36bfdcdfa0e549a65cbd2e"
server_key_version: "encryptionkey@0"
I20260929 20:26:40.462006 1474 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713339161492-17621-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713339161492-17621-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713339161492-17621-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:26:40.514493 1474 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:26:40.519076 1474 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:26:40.519352 1474 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:26:40.520437 1474 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:26:40.521881 1474 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:26:40.522001 1474 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:26:40.522146 1474 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:26:40.522236 1474 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:26:40.557516 1474 rpc_server.cc:307] RPC server started. Bound to: 127.17.53.66:33821
I20260929 20:26:40.557569 1601 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.53.66:33821 every 8 connection(s)
I20260929 20:26:40.559088 1474 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713339161492-17621-0/minicluster-data/ts-1/data/info.pb
I20260929 20:26:40.566845 17621 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu as pid 1474
I20260929 20:26:40.567776 17621 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
/tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713339161492-17621-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713339161492-17621-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713339161492-17621-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713339161492-17621-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.53.67:45491
--local_ip_for_outbound_sockets=127.17.53.67
--tserver_master_addrs=127.17.53.126:39625
--webserver_port=41667
--webserver_interface=127.17.53.67
--keytab_file=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.67.keytab
--principal=kudu/127.17.53.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.53.84:36709
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713339161492-17621-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713339161492-17621-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713339161492-17621-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 29 20:26:40 dist-test-slave-zbnc krb5kdc[699](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713600, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.53.66@KRBTEST.COM for kudu/127.17.53.126@KRBTEST.COM
I20260929 20:26:40.584168 1472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:26:40.561720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.53.66:52257 (local address 127.17.53.126:39625)
0929 20:26:40.561929 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0929 20:26:40.561935 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:26:40.563291 (+ 1356us) server_negotiation.cc:407] Connection header received
0929 20:26:40.565429 (+ 2138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:26:40.565435 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:26:40.565520 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:26:40.565609 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:26:40.568059 (+ 2450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:26:40.568768 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:26:40.571399 (+ 2631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:26:40.571816 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:26:40.571911 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:26:40.576569 (+ 4658us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:26:40.576578 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:26:40.576586 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:26:40.576642 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:26:40.580468 (+ 3826us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:26:40.581296 (+ 828us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:26:40.581305 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:26:40.581311 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:26:40.581507 (+ 196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:26:40.581965 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:26:40.581973 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:26:40.581979 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:26:40.582419 (+ 440us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:26:40.582609 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0929 20:26:40.583374 (+ 765us) server_negotiation.cc:299] Negotiation successful
0929 20:26:40.583666 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260929 20:26:40.584808 1604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:26:40.562207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:39625 (local address 127.17.53.66:52257)
0929 20:26:40.563104 (+ 897us) negotiation.cc:107] Waiting for socket to connect
0929 20:26:40.563161 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0929 20:26:40.565155 (+ 1994us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:26:40.565858 (+ 703us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:26:40.565866 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:26:40.566848 (+ 982us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:26:40.567777 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:26:40.567807 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:26:40.568964 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:26:40.568974 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:26:40.571249 (+ 2275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:26:40.571269 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:26:40.572097 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:26:40.572108 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:26:40.572461 (+ 353us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:26:40.573246 (+ 785us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:26:40.573301 (+ 55us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:26:40.576267 (+ 2966us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:26:40.580662 (+ 4395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:26:40.580669 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:26:40.580685 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:26:40.581138 (+ 453us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:26:40.581628 (+ 490us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:26:40.581632 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:26:40.581656 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:26:40.581838 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:26:40.582564 (+ 726us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:26:40.582570 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:26:40.582984 (+ 414us) client_negotiation.cc:769] Sending connection context
0929 20:26:40.583219 (+ 235us) client_negotiation.cc:240] Negotiation successful
0929 20:26:40.583626 (+ 407us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":523,"thread_start_us":303,"threads_started":1}
I20260929 20:26:40.586377 1602 heartbeater.cc:347] Connected to a master server at 127.17.53.126:39625
I20260929 20:26:40.586690 1602 heartbeater.cc:464] Registering TS with master...
I20260929 20:26:40.587539 1602 heartbeater.cc:511] Master 127.17.53.126:39625 requested a full tablet report, sending...
I20260929 20:26:40.589222 1284 ts_manager.cc:194] Registered new tserver with Master: 3b59fcbacafc43b088c6cacc664f6ccf (127.17.53.66:33821)
I20260929 20:26:40.590029 1284 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.53.66@KRBTEST.COM'} at 127.17.53.66:52257
W20260929 20:26:40.909330 1605 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:26:40.909623 1605 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:26:40.909781 1605 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:26:40.917131 1605 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:26:40.917248 1605 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:26:40.917440 1605 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.53.67
I20260929 20:26:40.927774 1605 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.53.84:36709
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713339161492-17621-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713339161492-17621-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.67.keytab
--principal=kudu/127.17.53.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.53.67:45491
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713339161492-17621-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.17.53.67
--webserver_port=41667
--tserver_master_addrs=127.17.53.126:39625
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.53.67
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713339161492-17621-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:26:40.929136 1605 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:26:40.930505 1605 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:26:40.944224 1610 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:26:40.944216 1611 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:26:40.944732 1613 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:26:40.945286 1605 server_base.cc:1034] running on GCE node
I20260929 20:26:40.945951 1605 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:26:40.947028 1605 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:26:40.948259 1605 hybrid_clock.cc:648] HybridClock initialized: now 1790713600948190 us; error 61 us; skew 500 ppm
I20260929 20:26:40.948597 1605 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:26:40 dist-test-slave-zbnc krb5kdc[699](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713600, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.53.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:26:40.951782 1605 init.cc:377] Logged in from keytab as kudu/127.17.53.67@KRBTEST.COM (short username kudu)
I20260929 20:26:40.953258 1605 webserver.cc:460] Webserver started at http://127.17.53.67:41667/ using document root <none> and password file <none>
I20260929 20:26:40.953956 1605 fs_manager.cc:359] Metadata directory not provided
I20260929 20:26:40.954115 1605 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:26:40.959827 1605 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.000s sys 0.004s
I20260929 20:26:40.962544 1620 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:26:40.964098 1605 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 20:26:40.964284 1605 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713339161492-17621-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713339161492-17621-0/minicluster-data/ts-2/wal
uuid: "445d9c75756f4730a2cb5bb98e8814e9"
format_stamp: "Formatted at 2026-09-29 20:26:18 on dist-test-slave-zbnc"
server_key: "730b50875bdd2ab9fe8dd6c354970414"
server_key_iv: "a91a7eb8fc8fda82009f81892050fd7d"
server_key_version: "encryptionkey@0"
I20260929 20:26:40.965279 1605 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713339161492-17621-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713339161492-17621-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713339161492-17621-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:26:40.977941 1605 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:26:40.982085 1605 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:26:40.982371 1605 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:26:40.983448 1605 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:26:40.984823 1605 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:26:40.984943 1605 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:26:40.985075 1605 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:26:40.985162 1605 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:26:41.020190 1605 rpc_server.cc:307] RPC server started. Bound to: 127.17.53.67:45491
I20260929 20:26:41.020280 1732 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.53.67:45491 every 8 connection(s)
I20260929 20:26:41.021941 1605 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713339161492-17621-0/minicluster-data/ts-2/data/info.pb
I20260929 20:26:41.027602 17621 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu as pid 1605
Sep 29 20:26:41 dist-test-slave-zbnc krb5kdc[699](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713600, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.53.67@KRBTEST.COM for kudu/127.17.53.126@KRBTEST.COM
I20260929 20:26:41.046470 1737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:26:41.028612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60864 (local address 127.17.53.126:39625)
0929 20:26:41.029367 (+ 755us) server_negotiation.cc:206] Beginning negotiation
0929 20:26:41.029373 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:26:41.029393 (+ 20us) server_negotiation.cc:407] Connection header received
0929 20:26:41.029446 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:26:41.029453 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:26:41.029532 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:26:41.029635 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:26:41.031688 (+ 2053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:26:41.032378 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:26:41.034196 (+ 1818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:26:41.034894 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:26:41.034995 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:26:41.038330 (+ 3335us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:26:41.038335 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:26:41.038340 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:26:41.038409 (+ 69us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:26:41.043157 (+ 4748us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:26:41.043983 (+ 826us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:26:41.043990 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:26:41.043993 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:26:41.044197 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:26:41.044719 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:26:41.044724 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:26:41.044726 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:26:41.045213 (+ 487us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:26:41.045494 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0929 20:26:41.045857 (+ 363us) server_negotiation.cc:299] Negotiation successful
0929 20:26:41.046033 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":128,"server-negotiator.queue_time_us":644,"spinlock_wait_cycles":31616,"thread_start_us":515,"threads_started":1}
I20260929 20:26:41.046931 1472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:26:41.024507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.53.67:52997 (local address 127.17.53.126:39625)
0929 20:26:41.024680 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0929 20:26:41.024684 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:26:41.025862 (+ 1178us) server_negotiation.cc:407] Connection header received
0929 20:26:41.027901 (+ 2039us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:26:41.027907 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:26:41.028026 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:26:41.028142 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:26:41.030298 (+ 2156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:26:41.031095 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:26:41.033018 (+ 1923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:26:41.033524 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:26:41.033613 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:26:41.038328 (+ 4715us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:26:41.038335 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:26:41.038340 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:26:41.038407 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:26:41.043068 (+ 4661us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:26:41.044004 (+ 936us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:26:41.044008 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:26:41.044010 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:26:41.044197 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:26:41.044728 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:26:41.044732 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:26:41.044735 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:26:41.045213 (+ 478us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:26:41.045372 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0929 20:26:41.046281 (+ 909us) server_negotiation.cc:299] Negotiation successful
0929 20:26:41.046494 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":61,"server-negotiator.queue_time_us":58,"spinlock_wait_cycles":14592}
I20260929 20:26:41.048326 1735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:26:41.025006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:39625 (local address 127.17.53.67:52997)
0929 20:26:41.025699 (+ 693us) negotiation.cc:107] Waiting for socket to connect
0929 20:26:41.025736 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0929 20:26:41.027572 (+ 1836us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:26:41.028440 (+ 868us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:26:41.028452 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:26:41.029302 (+ 850us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:26:41.030113 (+ 811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:26:41.030143 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:26:41.031258 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:26:41.031263 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:26:41.032880 (+ 1617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:26:41.032895 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:26:41.033743 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:26:41.033751 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:26:41.034076 (+ 325us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:26:41.034978 (+ 902us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:26:41.035010 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:26:41.038057 (+ 3047us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:26:41.043235 (+ 5178us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:26:41.043246 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:26:41.043265 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:26:41.043903 (+ 638us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:26:41.044323 (+ 420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:26:41.044327 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:26:41.044331 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:26:41.044631 (+ 300us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:26:41.045364 (+ 733us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:26:41.045372 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:26:41.045853 (+ 481us) client_negotiation.cc:769] Sending connection context
0929 20:26:41.046201 (+ 348us) client_negotiation.cc:240] Negotiation successful
0929 20:26:41.046805 (+ 604us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":275,"threads_started":1}
I20260929 20:26:41.049748 1733 heartbeater.cc:347] Connected to a master server at 127.17.53.126:39625
I20260929 20:26:41.050098 1733 heartbeater.cc:464] Registering TS with master...
I20260929 20:26:41.051136 1733 heartbeater.cc:511] Master 127.17.53.126:39625 requested a full tablet report, sending...
I20260929 20:26:41.053113 1284 ts_manager.cc:194] Registered new tserver with Master: 445d9c75756f4730a2cb5bb98e8814e9 (127.17.53.67:45491)
I20260929 20:26:41.054531 1284 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.53.67@KRBTEST.COM'} at 127.17.53.67:52997
I20260929 20:26:41.062700 17621 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 20:26:41.070199 1472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:26:41.063799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60866 (local address 127.17.53.126:39625)
0929 20:26:41.063970 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0929 20:26:41.063975 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:26:41.064468 (+ 493us) server_negotiation.cc:407] Connection header received
0929 20:26:41.064684 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:26:41.064689 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:26:41.064870 (+ 181us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:26:41.065106 (+ 236us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 20:26:41.065535 (+ 429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:26:41.066432 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:26:41.068271 (+ 1839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:26:41.068713 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:26:41.068823 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:26:41.069411 (+ 588us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 20:26:41.069677 (+ 266us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0929 20:26:41.069982 (+ 305us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:60866: token expired
Metrics: {"server-negotiator.queue_time_us":64}
W20260929 20:26:41.070344 1472 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:60866: token expired
W20260929 20:26:41.071044 1203 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.17.53.126:39625)
I20260929 20:26:41.084446 1472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:26:41.071572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60876 (local address 127.17.53.126:39625)
0929 20:26:41.071736 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0929 20:26:41.071741 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:26:41.071755 (+ 14us) server_negotiation.cc:407] Connection header received
0929 20:26:41.071936 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:26:41.071940 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:26:41.071995 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:26:41.072068 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:26:41.073479 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:26:41.074207 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:26:41.075649 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:26:41.076034 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:26:41.076122 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:26:41.079295 (+ 3173us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:26:41.079300 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:26:41.079303 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:26:41.079335 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:26:41.082002 (+ 2667us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:26:41.082742 (+ 740us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:26:41.082746 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:26:41.082748 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:26:41.082916 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:26:41.083267 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:26:41.083270 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:26:41.083272 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:26:41.083574 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:26:41.083783 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0929 20:26:41.083989 (+ 206us) server_negotiation.cc:299] Negotiation successful
0929 20:26:41.084160 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260929 20:26:41.088243 1284 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60876:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 20:26:41.096086 1284 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 20:26:41.096567 1470 heartbeater.cc:503] Master 127.17.53.126:39625 was elected leader, sending a full tablet report...
I20260929 20:26:41.118734 1746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:26:41.109584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38254 (local address 127.17.53.67:45491)
0929 20:26:41.110385 (+ 801us) server_negotiation.cc:206] Beginning negotiation
0929 20:26:41.110392 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:26:41.110420 (+ 28us) server_negotiation.cc:407] Connection header received
0929 20:26:41.110507 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:26:41.110512 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:26:41.110819 (+ 307us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:26:41.111714 (+ 895us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:26:41.111991 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:26:41.113614 (+ 1623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:26:41.116570 (+ 2956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:26:41.117503 (+ 933us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:26:41.117660 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:26:41.117904 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0929 20:26:41.118213 (+ 309us) server_negotiation.cc:299] Negotiation successful
0929 20:26:41.118373 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":602,"thread_start_us":1803,"threads_started":1}
I20260929 20:26:41.118804 1745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:26:41.108795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.67:45491 (local address 127.0.0.1:38254)
0929 20:26:41.110143 (+ 1348us) negotiation.cc:107] Waiting for socket to connect
0929 20:26:41.110160 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 20:26:41.110267 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:26:41.111073 (+ 806us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:26:41.111081 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:26:41.111111 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:26:41.111497 (+ 386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:26:41.111515 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:26:41.113758 (+ 2243us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:26:41.113764 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:26:41.116416 (+ 2652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:26:41.116428 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:26:41.117740 (+ 1312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:26:41.117746 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:26:41.117940 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:26:41.117988 (+ 48us) client_negotiation.cc:769] Sending connection context
0929 20:26:41.118177 (+ 189us) client_negotiation.cc:240] Negotiation successful
0929 20:26:41.118374 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1178,"mutex_wait_us":1,"thread_start_us":1140,"threads_started":1}
I20260929 20:26:41.120628 1747 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:26:41.110399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.65:43287 (local address 127.0.0.1:55252)
0929 20:26:41.110945 (+ 546us) negotiation.cc:107] Waiting for socket to connect
0929 20:26:41.110973 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0929 20:26:41.111071 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:26:41.112480 (+ 1409us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:26:41.112485 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:26:41.112513 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:26:41.113007 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:26:41.113021 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:26:41.115154 (+ 2133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:26:41.115159 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:26:41.117450 (+ 2291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:26:41.117469 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:26:41.119788 (+ 2319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:26:41.119795 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:26:41.119994 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:26:41.120027 (+ 33us) client_negotiation.cc:769] Sending connection context
0929 20:26:41.120096 (+ 69us) client_negotiation.cc:240] Negotiation successful
0929 20:26:41.120231 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":404,"mutex_wait_us":27,"thread_start_us":1211,"threads_started":1}
I20260929 20:26:41.121587 1748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:26:41.111180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55252 (local address 127.17.53.65:43287)
0929 20:26:41.111844 (+ 664us) server_negotiation.cc:206] Beginning negotiation
0929 20:26:41.111855 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 20:26:41.111881 (+ 26us) server_negotiation.cc:407] Connection header received
0929 20:26:41.111989 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:26:41.111997 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:26:41.112255 (+ 258us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:26:41.113220 (+ 965us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:26:41.113419 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:26:41.114990 (+ 1571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:26:41.117648 (+ 2658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:26:41.119535 (+ 1887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:26:41.120789 (+ 1254us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:26:41.121035 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0929 20:26:41.121110 (+ 75us) server_negotiation.cc:299] Negotiation successful
0929 20:26:41.121268 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":487,"thread_start_us":1569,"threads_started":1}
I20260929 20:26:41.121821 1743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:26:41.108484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.66:33821 (local address 127.0.0.1:48962)
0929 20:26:41.109490 (+ 1006us) negotiation.cc:107] Waiting for socket to connect
0929 20:26:41.109547 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0929 20:26:41.109710 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:26:41.113927 (+ 4217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:26:41.113932 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:26:41.113962 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:26:41.114374 (+ 412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:26:41.114389 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:26:41.116382 (+ 1993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:26:41.116388 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:26:41.118541 (+ 2153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:26:41.118558 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:26:41.121216 (+ 2658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:26:41.121222 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:26:41.121340 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:26:41.121364 (+ 24us) client_negotiation.cc:769] Sending connection context
0929 20:26:41.121414 (+ 50us) client_negotiation.cc:240] Negotiation successful
0929 20:26:41.121512 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":763,"thread_start_us":593,"threads_started":1}
I20260929 20:26:41.122781 1744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:26:41.108658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48962 (local address 127.17.53.66:33821)
0929 20:26:41.113418 (+ 4760us) server_negotiation.cc:206] Beginning negotiation
0929 20:26:41.113425 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:26:41.113448 (+ 23us) server_negotiation.cc:407] Connection header received
0929 20:26:41.113519 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:26:41.113523 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:26:41.113710 (+ 187us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:26:41.113930 (+ 220us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:26:41.114569 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:26:41.116235 (+ 1666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:26:41.119245 (+ 3010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:26:41.120932 (+ 1687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:26:41.122182 (+ 1250us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:26:41.122323 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0929 20:26:41.122374 (+ 51us) server_negotiation.cc:299] Negotiation successful
0929 20:26:41.122492 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4610,"thread_start_us":4501,"threads_started":1}
I20260929 20:26:41.123159 1668 tablet_service.cc:1467] Processing CreateTablet for tablet a4efb4834c074c13a954b83b14ddf289 (DEFAULT_TABLE table=test-table [id=8a5738b474384b3ab31c5050ec7ac1e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:26:41.124723 1668 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a4efb4834c074c13a954b83b14ddf289. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:26:41.125711 1405 tablet_service.cc:1467] Processing CreateTablet for tablet a4efb4834c074c13a954b83b14ddf289 (DEFAULT_TABLE table=test-table [id=8a5738b474384b3ab31c5050ec7ac1e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:26:41.126207 1537 tablet_service.cc:1467] Processing CreateTablet for tablet a4efb4834c074c13a954b83b14ddf289 (DEFAULT_TABLE table=test-table [id=8a5738b474384b3ab31c5050ec7ac1e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:26:41.127265 1405 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a4efb4834c074c13a954b83b14ddf289. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:26:41.127503 1537 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a4efb4834c074c13a954b83b14ddf289. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:26:41.138604 1749 tablet_bootstrap.cc:492] T a4efb4834c074c13a954b83b14ddf289 P 445d9c75756f4730a2cb5bb98e8814e9: Bootstrap starting.
I20260929 20:26:41.141959 1749 tablet_bootstrap.cc:654] T a4efb4834c074c13a954b83b14ddf289 P 445d9c75756f4730a2cb5bb98e8814e9: Neither blocks nor log segments found. Creating new log.
I20260929 20:26:41.143464 1749 log.cc:824] T a4efb4834c074c13a954b83b14ddf289 P 445d9c75756f4730a2cb5bb98e8814e9: Log is configured to *not* fsync() on all Append() calls
I20260929 20:26:41.144925 1751 tablet_bootstrap.cc:492] T a4efb4834c074c13a954b83b14ddf289 P 6e489a4674b840c4b61f170102698b7d: Bootstrap starting.
I20260929 20:26:41.146780 1749 tablet_bootstrap.cc:492] T a4efb4834c074c13a954b83b14ddf289 P 445d9c75756f4730a2cb5bb98e8814e9: No bootstrap required, opened a new log
I20260929 20:26:41.147073 1749 ts_tablet_manager.cc:1397] T a4efb4834c074c13a954b83b14ddf289 P 445d9c75756f4730a2cb5bb98e8814e9: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260929 20:26:41.147178 1750 tablet_bootstrap.cc:492] T a4efb4834c074c13a954b83b14ddf289 P 3b59fcbacafc43b088c6cacc664f6ccf: Bootstrap starting.
I20260929 20:26:41.148627 1751 tablet_bootstrap.cc:654] T a4efb4834c074c13a954b83b14ddf289 P 6e489a4674b840c4b61f170102698b7d: Neither blocks nor log segments found. Creating new log.
I20260929 20:26:41.149950 1751 log.cc:824] T a4efb4834c074c13a954b83b14ddf289 P 6e489a4674b840c4b61f170102698b7d: Log is configured to *not* fsync() on all Append() calls
I20260929 20:26:41.150123 1750 tablet_bootstrap.cc:654] T a4efb4834c074c13a954b83b14ddf289 P 3b59fcbacafc43b088c6cacc664f6ccf: Neither blocks nor log segments found. Creating new log.
I20260929 20:26:41.151419 1750 log.cc:824] T a4efb4834c074c13a954b83b14ddf289 P 3b59fcbacafc43b088c6cacc664f6ccf: Log is configured to *not* fsync() on all Append() calls
I20260929 20:26:41.153331 1751 tablet_bootstrap.cc:492] T a4efb4834c074c13a954b83b14ddf289 P 6e489a4674b840c4b61f170102698b7d: No bootstrap required, opened a new log
I20260929 20:26:41.153677 1751 ts_tablet_manager.cc:1397] T a4efb4834c074c13a954b83b14ddf289 P 6e489a4674b840c4b61f170102698b7d: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.008s
I20260929 20:26:41.154909 1750 tablet_bootstrap.cc:492] T a4efb4834c074c13a954b83b14ddf289 P 3b59fcbacafc43b088c6cacc664f6ccf: No bootstrap required, opened a new log
I20260929 20:26:41.155179 1750 ts_tablet_manager.cc:1397] T a4efb4834c074c13a954b83b14ddf289 P 3b59fcbacafc43b088c6cacc664f6ccf: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260929 20:26:41.156474 1749 raft_consensus.cc:348] T a4efb4834c074c13a954b83b14ddf289 P 445d9c75756f4730a2cb5bb98e8814e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b59fcbacafc43b088c6cacc664f6ccf" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 33821 } } peers { permanent_uuid: "445d9c75756f4730a2cb5bb98e8814e9" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 45491 } } peers { permanent_uuid: "6e489a4674b840c4b61f170102698b7d" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 43287 } }
I20260929 20:26:41.156844 1749 raft_consensus.cc:374] T a4efb4834c074c13a954b83b14ddf289 P 445d9c75756f4730a2cb5bb98e8814e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:26:41.156937 1749 raft_consensus.cc:729] T a4efb4834c074c13a954b83b14ddf289 P 445d9c75756f4730a2cb5bb98e8814e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 445d9c75756f4730a2cb5bb98e8814e9, State: Initialized, Role: FOLLOWER
I20260929 20:26:41.157546 1749 consensus_queue.cc:260] T a4efb4834c074c13a954b83b14ddf289 P 445d9c75756f4730a2cb5bb98e8814e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b59fcbacafc43b088c6cacc664f6ccf" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 33821 } } peers { permanent_uuid: "445d9c75756f4730a2cb5bb98e8814e9" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 45491 } } peers { permanent_uuid: "6e489a4674b840c4b61f170102698b7d" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 43287 } }
I20260929 20:26:41.158650 1733 heartbeater.cc:503] Master 127.17.53.126:39625 was elected leader, sending a full tablet report...
I20260929 20:26:41.159350 1749 ts_tablet_manager.cc:1428] T a4efb4834c074c13a954b83b14ddf289 P 445d9c75756f4730a2cb5bb98e8814e9: Time spent starting tablet: real 0.012s user 0.012s sys 0.000s
I20260929 20:26:41.164700 1751 raft_consensus.cc:348] T a4efb4834c074c13a954b83b14ddf289 P 6e489a4674b840c4b61f170102698b7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b59fcbacafc43b088c6cacc664f6ccf" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 33821 } } peers { permanent_uuid: "445d9c75756f4730a2cb5bb98e8814e9" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 45491 } } peers { permanent_uuid: "6e489a4674b840c4b61f170102698b7d" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 43287 } }
I20260929 20:26:41.165211 1751 raft_consensus.cc:374] T a4efb4834c074c13a954b83b14ddf289 P 6e489a4674b840c4b61f170102698b7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:26:41.165324 1751 raft_consensus.cc:729] T a4efb4834c074c13a954b83b14ddf289 P 6e489a4674b840c4b61f170102698b7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e489a4674b840c4b61f170102698b7d, State: Initialized, Role: FOLLOWER
I20260929 20:26:41.165904 1750 raft_consensus.cc:348] T a4efb4834c074c13a954b83b14ddf289 P 3b59fcbacafc43b088c6cacc664f6ccf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b59fcbacafc43b088c6cacc664f6ccf" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 33821 } } peers { permanent_uuid: "445d9c75756f4730a2cb5bb98e8814e9" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 45491 } } peers { permanent_uuid: "6e489a4674b840c4b61f170102698b7d" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 43287 } }
I20260929 20:26:41.166349 1750 raft_consensus.cc:374] T a4efb4834c074c13a954b83b14ddf289 P 3b59fcbacafc43b088c6cacc664f6ccf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:26:41.166158 1751 consensus_queue.cc:260] T a4efb4834c074c13a954b83b14ddf289 P 6e489a4674b840c4b61f170102698b7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b59fcbacafc43b088c6cacc664f6ccf" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 33821 } } peers { permanent_uuid: "445d9c75756f4730a2cb5bb98e8814e9" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 45491 } } peers { permanent_uuid: "6e489a4674b840c4b61f170102698b7d" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 43287 } }
I20260929 20:26:41.166450 1750 raft_consensus.cc:729] T a4efb4834c074c13a954b83b14ddf289 P 3b59fcbacafc43b088c6cacc664f6ccf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b59fcbacafc43b088c6cacc664f6ccf, State: Initialized, Role: FOLLOWER
I20260929 20:26:41.167132 1750 consensus_queue.cc:260] T a4efb4834c074c13a954b83b14ddf289 P 3b59fcbacafc43b088c6cacc664f6ccf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b59fcbacafc43b088c6cacc664f6ccf" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 33821 } } peers { permanent_uuid: "445d9c75756f4730a2cb5bb98e8814e9" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 45491 } } peers { permanent_uuid: "6e489a4674b840c4b61f170102698b7d" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 43287 } }
I20260929 20:26:41.168166 1751 ts_tablet_manager.cc:1428] T a4efb4834c074c13a954b83b14ddf289 P 6e489a4674b840c4b61f170102698b7d: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
I20260929 20:26:41.168248 1602 heartbeater.cc:503] Master 127.17.53.126:39625 was elected leader, sending a full tablet report...
I20260929 20:26:41.168891 1750 ts_tablet_manager.cc:1428] T a4efb4834c074c13a954b83b14ddf289 P 3b59fcbacafc43b088c6cacc664f6ccf: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
W20260929 20:26:41.273613 1734 tablet.cc:2369] T a4efb4834c074c13a954b83b14ddf289 P 445d9c75756f4730a2cb5bb98e8814e9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 20:26:41.282743 1756 raft_consensus.cc:482] T a4efb4834c074c13a954b83b14ddf289 P 6e489a4674b840c4b61f170102698b7d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 20:26:41.283135 1756 raft_consensus.cc:504] T a4efb4834c074c13a954b83b14ddf289 P 6e489a4674b840c4b61f170102698b7d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b59fcbacafc43b088c6cacc664f6ccf" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 33821 } } peers { permanent_uuid: "445d9c75756f4730a2cb5bb98e8814e9" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 45491 } } peers { permanent_uuid: "6e489a4674b840c4b61f170102698b7d" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 43287 } }
I20260929 20:26:41.284871 1756 leader_election.cc:288] T a4efb4834c074c13a954b83b14ddf289 P 6e489a4674b840c4b61f170102698b7d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3b59fcbacafc43b088c6cacc664f6ccf (127.17.53.66:33821), 445d9c75756f4730a2cb5bb98e8814e9 (127.17.53.67:45491)
I20260929 20:26:41.292311 1744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:26:41.285219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.53.65:36447 (local address 127.17.53.66:33821)
0929 20:26:41.285493 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0929 20:26:41.285503 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 20:26:41.286376 (+ 873us) server_negotiation.cc:407] Connection header received
0929 20:26:41.286554 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:26:41.286560 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:26:41.286642 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:26:41.286731 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:26:41.287443 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:26:41.288363 (+ 920us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:26:41.290389 (+ 2026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:26:41.291301 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:26:41.291479 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:26:41.291566 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0929 20:26:41.291887 (+ 321us) server_negotiation.cc:299] Negotiation successful
0929 20:26:41.292034 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260929 20:26:41.292313 1746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:26:41.285434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.53.65:48453 (local address 127.17.53.67:45491)
0929 20:26:41.285700 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0929 20:26:41.285708 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 20:26:41.286584 (+ 876us) server_negotiation.cc:407] Connection header received
0929 20:26:41.286766 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:26:41.286771 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:26:41.286848 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:26:41.286965 (+ 117us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:26:41.287553 (+ 588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:26:41.288365 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:26:41.290485 (+ 2120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:26:41.291310 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:26:41.291498 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:26:41.291577 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0929 20:26:41.291949 (+ 372us) server_negotiation.cc:299] Negotiation successful
0929 20:26:41.292069 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
I20260929 20:26:41.292446 1759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:26:41.285181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.67:45491 (local address 127.17.53.65:48453)
0929 20:26:41.286422 (+ 1241us) negotiation.cc:107] Waiting for socket to connect
0929 20:26:41.286469 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0929 20:26:41.286616 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:26:41.286980 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:26:41.286984 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:26:41.287005 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:26:41.287405 (+ 400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:26:41.287418 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:26:41.288503 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:26:41.288508 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:26:41.290332 (+ 1824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:26:41.290349 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:26:41.291577 (+ 1228us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:26:41.291586 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:26:41.291726 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:26:41.291794 (+ 68us) client_negotiation.cc:769] Sending connection context
0929 20:26:41.291909 (+ 115us) client_negotiation.cc:240] Negotiation successful
0929 20:26:41.292030 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1114,"mutex_wait_us":48,"thread_start_us":865,"threads_started":1}
I20260929 20:26:41.292860 1758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:26:41.285085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.66:33821 (local address 127.17.53.65:36447)
0929 20:26:41.286260 (+ 1175us) negotiation.cc:107] Waiting for socket to connect
0929 20:26:41.286277 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 20:26:41.286402 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:26:41.286761 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:26:41.286766 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:26:41.286791 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:26:41.287276 (+ 485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:26:41.287290 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:26:41.288496 (+ 1206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:26:41.288502 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:26:41.290232 (+ 1730us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:26:41.290245 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:26:41.291557 (+ 1312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:26:41.291564 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:26:41.291716 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:26:41.291756 (+ 40us) client_negotiation.cc:769] Sending connection context
0929 20:26:41.291830 (+ 74us) client_negotiation.cc:240] Negotiation successful
0929 20:26:41.291971 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1035,"mutex_wait_us":2,"thread_start_us":809,"threads_started":1}
I20260929 20:26:41.293533 1688 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4efb4834c074c13a954b83b14ddf289" candidate_uuid: "6e489a4674b840c4b61f170102698b7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "445d9c75756f4730a2cb5bb98e8814e9" is_pre_election: true
I20260929 20:26:41.293841 1557 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4efb4834c074c13a954b83b14ddf289" candidate_uuid: "6e489a4674b840c4b61f170102698b7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b59fcbacafc43b088c6cacc664f6ccf" is_pre_election: true
I20260929 20:26:41.294066 1688 raft_consensus.cc:2454] T a4efb4834c074c13a954b83b14ddf289 P 445d9c75756f4730a2cb5bb98e8814e9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e489a4674b840c4b61f170102698b7d in term 0.
I20260929 20:26:41.294320 1557 raft_consensus.cc:2454] T a4efb4834c074c13a954b83b14ddf289 P 3b59fcbacafc43b088c6cacc664f6ccf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6e489a4674b840c4b61f170102698b7d in term 0.
I20260929 20:26:41.294804 1359 leader_election.cc:302] T a4efb4834c074c13a954b83b14ddf289 P 6e489a4674b840c4b61f170102698b7d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 445d9c75756f4730a2cb5bb98e8814e9, 6e489a4674b840c4b61f170102698b7d; no voters:
I20260929 20:26:41.295243 1756 raft_consensus.cc:2781] T a4efb4834c074c13a954b83b14ddf289 P 6e489a4674b840c4b61f170102698b7d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 20:26:41.295406 1756 raft_consensus.cc:482] T a4efb4834c074c13a954b83b14ddf289 P 6e489a4674b840c4b61f170102698b7d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 20:26:41.295569 1756 raft_consensus.cc:3037] T a4efb4834c074c13a954b83b14ddf289 P 6e489a4674b840c4b61f170102698b7d [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:26:41.298096 1756 raft_consensus.cc:504] T a4efb4834c074c13a954b83b14ddf289 P 6e489a4674b840c4b61f170102698b7d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b59fcbacafc43b088c6cacc664f6ccf" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 33821 } } peers { permanent_uuid: "445d9c75756f4730a2cb5bb98e8814e9" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 45491 } } peers { permanent_uuid: "6e489a4674b840c4b61f170102698b7d" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 43287 } }
I20260929 20:26:41.298708 1756 leader_election.cc:288] T a4efb4834c074c13a954b83b14ddf289 P 6e489a4674b840c4b61f170102698b7d [CANDIDATE]: Term 1 election: Requested vote from peers 3b59fcbacafc43b088c6cacc664f6ccf (127.17.53.66:33821), 445d9c75756f4730a2cb5bb98e8814e9 (127.17.53.67:45491)
I20260929 20:26:41.299182 1557 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4efb4834c074c13a954b83b14ddf289" candidate_uuid: "6e489a4674b840c4b61f170102698b7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b59fcbacafc43b088c6cacc664f6ccf"
I20260929 20:26:41.299188 1688 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4efb4834c074c13a954b83b14ddf289" candidate_uuid: "6e489a4674b840c4b61f170102698b7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "445d9c75756f4730a2cb5bb98e8814e9"
I20260929 20:26:41.299391 1557 raft_consensus.cc:3037] T a4efb4834c074c13a954b83b14ddf289 P 3b59fcbacafc43b088c6cacc664f6ccf [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:26:41.299394 1688 raft_consensus.cc:3037] T a4efb4834c074c13a954b83b14ddf289 P 445d9c75756f4730a2cb5bb98e8814e9 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:26:41.301642 1688 raft_consensus.cc:2454] T a4efb4834c074c13a954b83b14ddf289 P 445d9c75756f4730a2cb5bb98e8814e9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e489a4674b840c4b61f170102698b7d in term 1.
I20260929 20:26:41.301643 1557 raft_consensus.cc:2454] T a4efb4834c074c13a954b83b14ddf289 P 3b59fcbacafc43b088c6cacc664f6ccf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6e489a4674b840c4b61f170102698b7d in term 1.
I20260929 20:26:41.302047 1359 leader_election.cc:302] T a4efb4834c074c13a954b83b14ddf289 P 6e489a4674b840c4b61f170102698b7d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 445d9c75756f4730a2cb5bb98e8814e9, 6e489a4674b840c4b61f170102698b7d; no voters:
I20260929 20:26:41.302315 1756 raft_consensus.cc:2781] T a4efb4834c074c13a954b83b14ddf289 P 6e489a4674b840c4b61f170102698b7d [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:26:41.302846 1756 raft_consensus.cc:686] T a4efb4834c074c13a954b83b14ddf289 P 6e489a4674b840c4b61f170102698b7d [term 1 LEADER]: Becoming Leader. State: Replica: 6e489a4674b840c4b61f170102698b7d, State: Running, Role: LEADER
I20260929 20:26:41.303440 1756 consensus_queue.cc:237] T a4efb4834c074c13a954b83b14ddf289 P 6e489a4674b840c4b61f170102698b7d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b59fcbacafc43b088c6cacc664f6ccf" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 33821 } } peers { permanent_uuid: "445d9c75756f4730a2cb5bb98e8814e9" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 45491 } } peers { permanent_uuid: "6e489a4674b840c4b61f170102698b7d" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 43287 } }
W20260929 20:26:41.305126 1471 tablet.cc:2369] T a4efb4834c074c13a954b83b14ddf289 P 6e489a4674b840c4b61f170102698b7d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 20:26:41.308058 1284 catalog_manager.cc:5495] T a4efb4834c074c13a954b83b14ddf289 P 6e489a4674b840c4b61f170102698b7d reported cstate change: term changed from 0 to 1, leader changed from <none> to 6e489a4674b840c4b61f170102698b7d (127.17.53.65). New cstate: current_term: 1 leader_uuid: "6e489a4674b840c4b61f170102698b7d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3b59fcbacafc43b088c6cacc664f6ccf" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 33821 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "445d9c75756f4730a2cb5bb98e8814e9" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 45491 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6e489a4674b840c4b61f170102698b7d" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 43287 } health_report { overall_health: HEALTHY } } }
W20260929 20:26:41.311252 1603 tablet.cc:2369] T a4efb4834c074c13a954b83b14ddf289 P 3b59fcbacafc43b088c6cacc664f6ccf: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 20:26:41.363185 1748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:26:41.357035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55268 (local address 127.17.53.65:43287)
0929 20:26:41.357277 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0929 20:26:41.357283 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:26:41.357408 (+ 125us) server_negotiation.cc:407] Connection header received
0929 20:26:41.357683 (+ 275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:26:41.357687 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:26:41.357763 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:26:41.357922 (+ 159us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 20:26:41.358304 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:26:41.359084 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:26:41.360552 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:26:41.360968 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:26:41.361102 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:26:41.361648 (+ 546us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 20:26:41.362431 (+ 783us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 20:26:41.362629 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0929 20:26:41.362789 (+ 160us) server_negotiation.cc:299] Negotiation successful
0929 20:26:41.362903 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260929 20:26:41.372812 1688 raft_consensus.cc:1261] T a4efb4834c074c13a954b83b14ddf289 P 445d9c75756f4730a2cb5bb98e8814e9 [term 1 FOLLOWER]: Refusing update from remote peer 6e489a4674b840c4b61f170102698b7d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:26:41.372838 1557 raft_consensus.cc:1261] T a4efb4834c074c13a954b83b14ddf289 P 3b59fcbacafc43b088c6cacc664f6ccf [term 1 FOLLOWER]: Refusing update from remote peer 6e489a4674b840c4b61f170102698b7d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:26:41.373848 1760 consensus_queue.cc:1035] T a4efb4834c074c13a954b83b14ddf289 P 6e489a4674b840c4b61f170102698b7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "445d9c75756f4730a2cb5bb98e8814e9" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 45491 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:26:41.374222 1756 consensus_queue.cc:1035] T a4efb4834c074c13a954b83b14ddf289 P 6e489a4674b840c4b61f170102698b7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b59fcbacafc43b088c6cacc664f6ccf" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 33821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:26:41.403458 1284 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60876:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 20:26:41.403774 1284 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60876:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 20:26:41.407392 1284 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e5f2bfb42723428996a3b2e3bb46f683: Sending DeleteTablet for 3 replicas of tablet a4efb4834c074c13a954b83b14ddf289
I20260929 20:26:41.408589 1537 tablet_service.cc:1514] Processing DeleteTablet for tablet a4efb4834c074c13a954b83b14ddf289 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:26:41 UTC) from {username='kudu', principal='kudu/127.17.53.126@KRBTEST.COM'} at 127.0.0.1:48962
I20260929 20:26:41.408691 1405 tablet_service.cc:1514] Processing DeleteTablet for tablet a4efb4834c074c13a954b83b14ddf289 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:26:41 UTC) from {username='kudu', principal='kudu/127.17.53.126@KRBTEST.COM'} at 127.0.0.1:55252
I20260929 20:26:41.409369 1668 tablet_service.cc:1514] Processing DeleteTablet for tablet a4efb4834c074c13a954b83b14ddf289 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:26:41 UTC) from {username='kudu', principal='kudu/127.17.53.126@KRBTEST.COM'} at 127.0.0.1:38254
I20260929 20:26:41.409655 1773 tablet_replica.cc:330] T a4efb4834c074c13a954b83b14ddf289 P 3b59fcbacafc43b088c6cacc664f6ccf: stopping tablet replica
I20260929 20:26:41.410166 1773 raft_consensus.cc:2229] T a4efb4834c074c13a954b83b14ddf289 P 3b59fcbacafc43b088c6cacc664f6ccf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 20:26:41.410501 17621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu pid 1321
I20260929 20:26:41.410903 1773 raft_consensus.cc:2258] T a4efb4834c074c13a954b83b14ddf289 P 3b59fcbacafc43b088c6cacc664f6ccf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:26:41.413107 1773 ts_tablet_manager.cc:1905] T a4efb4834c074c13a954b83b14ddf289 P 3b59fcbacafc43b088c6cacc664f6ccf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:26:41.419122 1774 tablet_replica.cc:330] T a4efb4834c074c13a954b83b14ddf289 P 6e489a4674b840c4b61f170102698b7d: stopping tablet replica
I20260929 20:26:41.419211 1773 ts_tablet_manager.cc:1918] T a4efb4834c074c13a954b83b14ddf289 P 3b59fcbacafc43b088c6cacc664f6ccf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 20:26:41.419498 1773 log.cc:1196] T a4efb4834c074c13a954b83b14ddf289 P 3b59fcbacafc43b088c6cacc664f6ccf: Deleting WAL directory at /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713339161492-17621-0/minicluster-data/ts-1/wal/wals/a4efb4834c074c13a954b83b14ddf289
I20260929 20:26:41.419764 1774 raft_consensus.cc:2229] T a4efb4834c074c13a954b83b14ddf289 P 6e489a4674b840c4b61f170102698b7d [term 1 LEADER]: Raft consensus shutting down.
I20260929 20:26:41.420171 1773 ts_tablet_manager.cc:1939] T a4efb4834c074c13a954b83b14ddf289 P 3b59fcbacafc43b088c6cacc664f6ccf: Deleting consensus metadata
I20260929 20:26:41.420611 1774 raft_consensus.cc:2258] T a4efb4834c074c13a954b83b14ddf289 P 6e489a4674b840c4b61f170102698b7d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:26:41.422330 1268 catalog_manager.cc:4841] TS 3b59fcbacafc43b088c6cacc664f6ccf (127.17.53.66:33821): tablet a4efb4834c074c13a954b83b14ddf289 (table test-table [id=8a5738b474384b3ab31c5050ec7ac1e2]) successfully deleted
I20260929 20:26:41.423370 1774 ts_tablet_manager.cc:1905] T a4efb4834c074c13a954b83b14ddf289 P 6e489a4674b840c4b61f170102698b7d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:26:41.424093 1775 tablet_replica.cc:330] T a4efb4834c074c13a954b83b14ddf289 P 445d9c75756f4730a2cb5bb98e8814e9: stopping tablet replica
I20260929 20:26:41.424564 1775 raft_consensus.cc:2229] T a4efb4834c074c13a954b83b14ddf289 P 445d9c75756f4730a2cb5bb98e8814e9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 20:26:41.425123 1775 raft_consensus.cc:2258] T a4efb4834c074c13a954b83b14ddf289 P 445d9c75756f4730a2cb5bb98e8814e9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:26:41.427641 1775 ts_tablet_manager.cc:1905] T a4efb4834c074c13a954b83b14ddf289 P 445d9c75756f4730a2cb5bb98e8814e9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:26:41.431162 1748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:26:41.412164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55284 (local address 127.17.53.65:43287)
0929 20:26:41.412358 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0929 20:26:41.412365 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:26:41.412386 (+ 21us) server_negotiation.cc:407] Connection header received
0929 20:26:41.412447 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:26:41.412452 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:26:41.412544 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:26:41.412649 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:26:41.414473 (+ 1824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:26:41.415799 (+ 1326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:26:41.417733 (+ 1934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:26:41.418455 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:26:41.418582 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:26:41.423635 (+ 5053us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:26:41.423651 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:26:41.423672 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:26:41.423729 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:26:41.428073 (+ 4344us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:26:41.428824 (+ 751us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:26:41.428833 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:26:41.428837 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:26:41.429046 (+ 209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:26:41.429429 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:26:41.429434 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:26:41.429437 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:26:41.429859 (+ 422us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:26:41.430166 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0929 20:26:41.430487 (+ 321us) server_negotiation.cc:299] Negotiation successful
0929 20:26:41.430754 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260929 20:26:41.432190 1465 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:26:41.436388 1775 ts_tablet_manager.cc:1918] T a4efb4834c074c13a954b83b14ddf289 P 445d9c75756f4730a2cb5bb98e8814e9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 20:26:41.436573 1775 log.cc:1196] T a4efb4834c074c13a954b83b14ddf289 P 445d9c75756f4730a2cb5bb98e8814e9: Deleting WAL directory at /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713339161492-17621-0/minicluster-data/ts-2/wal/wals/a4efb4834c074c13a954b83b14ddf289
I20260929 20:26:41.437021 1775 ts_tablet_manager.cc:1939] T a4efb4834c074c13a954b83b14ddf289 P 445d9c75756f4730a2cb5bb98e8814e9: Deleting consensus metadata
I20260929 20:26:41.439443 1269 catalog_manager.cc:4841] TS 445d9c75756f4730a2cb5bb98e8814e9 (127.17.53.67:45491): tablet a4efb4834c074c13a954b83b14ddf289 (table test-table [id=8a5738b474384b3ab31c5050ec7ac1e2]) successfully deleted
I20260929 20:26:41.535668 17621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu with pid 1321
I20260929 20:26:41.554831 17621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu pid 1474
W20260929 20:26:41.554996 1269 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.17.53.65:43287 (error 108)
I20260929 20:26:41.556906 1743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:26:41.556367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.65:43287 (local address 127.0.0.1:55290)
0929 20:26:41.556592 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0929 20:26:41.556724 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.53.65:43287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
W20260929 20:26:41.558589 1269 catalog_manager.cc:4584] TS 6e489a4674b840c4b61f170102698b7d (127.17.53.65:43287): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a4efb4834c074c13a954b83b14ddf289: Network error: Client connection negotiation failed: client connection to 127.17.53.65:43287: connect: Connection refused (error 111)
I20260929 20:26:41.570398 1744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:26:41.555815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48974 (local address 127.17.53.66:33821)
0929 20:26:41.556387 (+ 572us) server_negotiation.cc:206] Beginning negotiation
0929 20:26:41.556391 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:26:41.556433 (+ 42us) server_negotiation.cc:407] Connection header received
0929 20:26:41.556513 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:26:41.556518 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:26:41.556609 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:26:41.556713 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:26:41.558440 (+ 1727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:26:41.559641 (+ 1201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:26:41.560947 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:26:41.561355 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:26:41.561498 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:26:41.564186 (+ 2688us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:26:41.564197 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:26:41.564209 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:26:41.564247 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:26:41.567447 (+ 3200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:26:41.568145 (+ 698us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:26:41.568152 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:26:41.568154 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:26:41.568320 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:26:41.568726 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:26:41.568732 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:26:41.568735 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:26:41.569165 (+ 430us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:26:41.569470 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0929 20:26:41.569810 (+ 340us) server_negotiation.cc:299] Negotiation successful
0929 20:26:41.570057 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":461}
I20260929 20:26:41.571163 1597 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:26:41.612417 1743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:26:41.611934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.65:43287 (local address 127.0.0.1:55300)
0929 20:26:41.612148 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0929 20:26:41.612260 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.53.65:43287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260929 20:26:41.613468 1743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:26:41.613169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.65:43287 (local address 127.0.0.1:55314)
0929 20:26:41.613297 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0929 20:26:41.613365 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.53.65:43287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260929 20:26:41.663471 17621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu with pid 1474
I20260929 20:26:41.673938 1743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:26:41.673469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.65:43287 (local address 127.0.0.1:55316)
0929 20:26:41.673697 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0929 20:26:41.673787 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.53.65:43287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260929 20:26:41.675021 1743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:26:41.674622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.65:43287 (local address 127.0.0.1:55322)
0929 20:26:41.674772 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0929 20:26:41.674850 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.53.65:43287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260929 20:26:41.681421 17621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu pid 1605
I20260929 20:26:41.695659 1746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:26:41.682197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38270 (local address 127.17.53.67:45491)
0929 20:26:41.682358 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0929 20:26:41.682363 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:26:41.682380 (+ 17us) server_negotiation.cc:407] Connection header received
0929 20:26:41.682457 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:26:41.682460 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:26:41.682528 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:26:41.682607 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:26:41.684035 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:26:41.684771 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:26:41.686056 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:26:41.686433 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:26:41.686526 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:26:41.689311 (+ 2785us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:26:41.689326 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:26:41.689343 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:26:41.689387 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:26:41.692632 (+ 3245us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:26:41.693370 (+ 738us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:26:41.693378 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:26:41.693381 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:26:41.693536 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:26:41.693930 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:26:41.693994 (+ 64us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:26:41.693997 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:26:41.694376 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:26:41.694605 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0929 20:26:41.695200 (+ 595us) server_negotiation.cc:299] Negotiation successful
0929 20:26:41.695372 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 20:26:41.696327 1728 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:26:41.756016 1743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:26:41.755365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.65:43287 (local address 127.0.0.1:55328)
0929 20:26:41.755676 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0929 20:26:41.755796 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.53.65:43287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110}
I20260929 20:26:41.757175 1743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:26:41.756750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.65:43287 (local address 127.0.0.1:55330)
0929 20:26:41.756962 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0929 20:26:41.757049 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.53.65:43287: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111}
I20260929 20:26:41.786413 17621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu with pid 1605
I20260929 20:26:41.807374 17621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu pid 1251
I20260929 20:26:41.808665 1314 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:26:41.928712 17621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu with pid 1251
I20260929 20:27:01.953037 17621 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
/tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713339161492-17621-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713339161492-17621-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713339161492-17621-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713339161492-17621-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.53.126:39625
--webserver_interface=127.17.53.126
--webserver_port=42051
--keytab_file=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.126.keytab
--principal=kudu/127.17.53.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.53.84:36709
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.53.126:39625
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713339161492-17621-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713339161492-17621-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713339161492-17621-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:27:02.295866 1781 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:27:02.296186 1781 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:27:02.296393 1781 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:27:02.303732 1781 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260929 20:27:02.303843 1781 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260929 20:27:02.303920 1781 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260929 20:27:02.304000 1781 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:27:02.304087 1781 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:27:02.304206 1781 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:27:02.304286 1781 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:27:02.304368 1781 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:27:02.314607 1781 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.53.84:36709
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713339161492-17621-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713339161492-17621-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.17.53.126:39625
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.126.keytab
--principal=kudu/127.17.53.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.53.126:39625
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713339161492-17621-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.53.126
--webserver_port=42051
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713339161492-17621-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:27:02.316071 1781 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:27:02.317488 1781 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:27:02.330209 1787 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:27:02.330179 1789 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:27:02.330258 1786 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:27:02.332744 1781 server_base.cc:1034] running on GCE node
I20260929 20:27:02.333714 1781 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:27:02.335588 1781 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:27:02.336874 1781 hybrid_clock.cc:648] HybridClock initialized: now 1790713622336807 us; error 42 us; skew 500 ppm
I20260929 20:27:02.337301 1781 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:27:02 dist-test-slave-zbnc krb5kdc[699](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713622, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.53.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:27:02.348248 1781 init.cc:377] Logged in from keytab as kudu/127.17.53.126@KRBTEST.COM (short username kudu)
I20260929 20:27:02.349975 1781 webserver.cc:460] Webserver started at http://127.17.53.126:42051/ using document root <none> and password file <none>
I20260929 20:27:02.350793 1781 fs_manager.cc:359] Metadata directory not provided
I20260929 20:27:02.350958 1781 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:27:02.357692 1781 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 20:27:02.360847 1796 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:27:02.362475 1781 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 20:27:02.362700 1781 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713339161492-17621-0/minicluster-data/master-0/data,/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713339161492-17621-0/minicluster-data/master-0/wal
uuid: "e5f2bfb42723428996a3b2e3bb46f683"
format_stamp: "Formatted at 2026-09-29 20:26:16 on dist-test-slave-zbnc"
server_key: "6f9bd922439cf9858a42a8d6ec9ce281"
server_key_iv: "dcf1d40a474d3e29ab4ba8f68c56b511"
server_key_version: "encryptionkey@0"
I20260929 20:27:02.363850 1781 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713339161492-17621-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713339161492-17621-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713339161492-17621-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:27:02.391018 1781 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:27:02.395740 1781 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:27:02.396073 1781 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:27:02.415904 1781 rpc_server.cc:307] RPC server started. Bound to: 127.17.53.126:39625
I20260929 20:27:02.415930 1847 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.53.126:39625 every 8 connection(s)
I20260929 20:27:02.417549 1781 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713339161492-17621-0/minicluster-data/master-0/data/info.pb
I20260929 20:27:02.418457 17621 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu as pid 1781
I20260929 20:27:02.419513 17621 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
/tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713339161492-17621-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713339161492-17621-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713339161492-17621-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713339161492-17621-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.53.65:43287
--local_ip_for_outbound_sockets=127.17.53.65
--tserver_master_addrs=127.17.53.126:39625
--webserver_port=33787
--webserver_interface=127.17.53.65
--keytab_file=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.65.keytab
--principal=kudu/127.17.53.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.53.84:36709
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713339161492-17621-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713339161492-17621-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713339161492-17621-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 20:27:02.425830 1848 sys_catalog.cc:263] Verifying existing consensus state
I20260929 20:27:02.429561 1848 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e5f2bfb42723428996a3b2e3bb46f683: Bootstrap starting.
I20260929 20:27:02.452728 1848 log.cc:824] T 00000000000000000000000000000000 P e5f2bfb42723428996a3b2e3bb46f683: Log is configured to *not* fsync() on all Append() calls
I20260929 20:27:02.469923 1848 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e5f2bfb42723428996a3b2e3bb46f683: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 20:27:02.470518 1848 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e5f2bfb42723428996a3b2e3bb46f683: Bootstrap complete.
I20260929 20:27:02.479985 1848 raft_consensus.cc:348] T 00000000000000000000000000000000 P e5f2bfb42723428996a3b2e3bb46f683 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5f2bfb42723428996a3b2e3bb46f683" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 39625 } }
I20260929 20:27:02.480890 1848 raft_consensus.cc:729] T 00000000000000000000000000000000 P e5f2bfb42723428996a3b2e3bb46f683 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5f2bfb42723428996a3b2e3bb46f683, State: Initialized, Role: FOLLOWER
I20260929 20:27:02.481595 1848 consensus_queue.cc:260] T 00000000000000000000000000000000 P e5f2bfb42723428996a3b2e3bb46f683 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5f2bfb42723428996a3b2e3bb46f683" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 39625 } }
I20260929 20:27:02.481844 1848 raft_consensus.cc:388] T 00000000000000000000000000000000 P e5f2bfb42723428996a3b2e3bb46f683 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 20:27:02.481971 1848 raft_consensus.cc:482] T 00000000000000000000000000000000 P e5f2bfb42723428996a3b2e3bb46f683 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 20:27:02.482160 1848 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e5f2bfb42723428996a3b2e3bb46f683 [term 2 FOLLOWER]: Advancing to term 3
I20260929 20:27:02.487106 1848 raft_consensus.cc:504] T 00000000000000000000000000000000 P e5f2bfb42723428996a3b2e3bb46f683 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5f2bfb42723428996a3b2e3bb46f683" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 39625 } }
I20260929 20:27:02.488071 1848 leader_election.cc:302] T 00000000000000000000000000000000 P e5f2bfb42723428996a3b2e3bb46f683 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e5f2bfb42723428996a3b2e3bb46f683; no voters:
I20260929 20:27:02.489185 1848 leader_election.cc:288] T 00000000000000000000000000000000 P e5f2bfb42723428996a3b2e3bb46f683 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260929 20:27:02.489295 1852 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e5f2bfb42723428996a3b2e3bb46f683 [term 3 FOLLOWER]: Leader election won for term 3
I20260929 20:27:02.492004 1848 sys_catalog.cc:564] T 00000000000000000000000000000000 P e5f2bfb42723428996a3b2e3bb46f683 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:27:02.494947 1852 raft_consensus.cc:686] T 00000000000000000000000000000000 P e5f2bfb42723428996a3b2e3bb46f683 [term 3 LEADER]: Becoming Leader. State: Replica: e5f2bfb42723428996a3b2e3bb46f683, State: Running, Role: LEADER
I20260929 20:27:02.495702 1852 consensus_queue.cc:237] T 00000000000000000000000000000000 P e5f2bfb42723428996a3b2e3bb46f683 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5f2bfb42723428996a3b2e3bb46f683" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 39625 } }
I20260929 20:27:02.498728 1853 sys_catalog.cc:455] T 00000000000000000000000000000000 P e5f2bfb42723428996a3b2e3bb46f683 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e5f2bfb42723428996a3b2e3bb46f683. Latest consensus state: current_term: 3 leader_uuid: "e5f2bfb42723428996a3b2e3bb46f683" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5f2bfb42723428996a3b2e3bb46f683" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 39625 } } }
I20260929 20:27:02.499195 1853 sys_catalog.cc:458] T 00000000000000000000000000000000 P e5f2bfb42723428996a3b2e3bb46f683 [sys.catalog]: This master's current role is: LEADER
I20260929 20:27:02.501987 1854 sys_catalog.cc:455] T 00000000000000000000000000000000 P e5f2bfb42723428996a3b2e3bb46f683 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "e5f2bfb42723428996a3b2e3bb46f683" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e5f2bfb42723428996a3b2e3bb46f683" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 39625 } } }
I20260929 20:27:02.502370 1854 sys_catalog.cc:458] T 00000000000000000000000000000000 P e5f2bfb42723428996a3b2e3bb46f683 [sys.catalog]: This master's current role is: LEADER
I20260929 20:27:02.502590 1858 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:27:02.518440 1858 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:27:02.525842 1858 catalog_manager.cc:1259] Loaded cluster ID: 440651dfe5e2435689788eaff7deaa18
I20260929 20:27:02.527562 1858 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:27:02.532250 1858 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:27:02.538892 1858 catalog_manager.cc:5879] T 00000000000000000000000000000000 P e5f2bfb42723428996a3b2e3bb46f683: Loaded TSK: 0
I20260929 20:27:02.539945 1858 catalog_manager.cc:5879] T 00000000000000000000000000000000 P e5f2bfb42723428996a3b2e3bb46f683: Loaded TSK: 1
F20260929 20:27:02.552502 1858 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f516b1389ad google::LogMessage::Fail() at ??:0
@ 0x7f516b13b88a google::LogMessage::SendToLog() at ??:0
@ 0x7f516b1384de google::LogMessage::Flush() at ??:0
@ 0x7f516b139a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f516ee9c7d2 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f516ee8b73e kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f5180b6b1c0 kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f5180b6a783 kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f5180bf62a7 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_10::operator()() at ??:0
@ 0x7f5180bf60c4 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7f5180c405e9 std::function<>::operator()() at ??:0
@ 0x7f5180b6ecdd kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f5180b6cc36 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f5180bf4610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f5180bf4439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f516dd84345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f516dd8791c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f516dd87799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f516dd62eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f517103c6db start_thread at ??:0
@ 0x7f5169fd261f clone at ??:0
W20260929 20:27:02.840811 1850 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:27:02.841135 1850 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:27:02.841272 1850 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:27:02.848652 1850 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:27:02.848757 1850 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:27:02.848869 1850 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.53.65
I20260929 20:27:02.858806 1850 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.53.84:36709
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713339161492-17621-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713339161492-17621-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.65.keytab
--principal=kudu/127.17.53.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.53.65:43287
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713339161492-17621-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.17.53.65
--webserver_port=33787
--tserver_master_addrs=127.17.53.126:39625
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.53.65
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713339161492-17621-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:27:02.860152 1850 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:27:02.861497 1850 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:27:02.875238 1897 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:27:02.875303 1899 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:27:02.875291 1896 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:27:02.876698 1850 server_base.cc:1034] running on GCE node
I20260929 20:27:02.877245 1850 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:27:02.878165 1850 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:27:02.879371 1850 hybrid_clock.cc:648] HybridClock initialized: now 1790713622879326 us; error 35 us; skew 500 ppm
I20260929 20:27:02.879701 1850 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:27:02 dist-test-slave-zbnc krb5kdc[699](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713622, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.53.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:27:02.882830 1850 init.cc:377] Logged in from keytab as kudu/127.17.53.65@KRBTEST.COM (short username kudu)
I20260929 20:27:02.884378 1850 webserver.cc:460] Webserver started at http://127.17.53.65:33787/ using document root <none> and password file <none>
I20260929 20:27:02.885092 1850 fs_manager.cc:359] Metadata directory not provided
I20260929 20:27:02.885241 1850 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:27:02.890743 1850 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260929 20:27:02.893711 1906 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:27:02.895253 1850 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 20:27:02.895429 1850 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713339161492-17621-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713339161492-17621-0/minicluster-data/ts-0/wal
uuid: "6e489a4674b840c4b61f170102698b7d"
format_stamp: "Formatted at 2026-09-29 20:26:17 on dist-test-slave-zbnc"
server_key: "5684d05b4cfcfc1fb7d6fa54ad8e7c7b"
server_key_iv: "c257d9544ebce464fb7d5810b92342c8"
server_key_version: "encryptionkey@0"
I20260929 20:27:02.896377 1850 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713339161492-17621-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713339161492-17621-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713339161492-17621-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:27:02.921861 1850 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:27:02.926090 1850 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:27:02.926388 1850 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:27:02.927459 1850 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:27:02.929322 1913 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
W20260929 20:27:02.934703 1913 ts_tablet_manager.cc:1852] T a4efb4834c074c13a954b83b14ddf289 P 6e489a4674b840c4b61f170102698b7d: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260929 20:27:02.935766 1913 ts_tablet_manager.cc:1905] T a4efb4834c074c13a954b83b14ddf289 P 6e489a4674b840c4b61f170102698b7d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:27:02.943032 1913 ts_tablet_manager.cc:1918] T a4efb4834c074c13a954b83b14ddf289 P 6e489a4674b840c4b61f170102698b7d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 20:27:02.943207 1913 log.cc:1196] T a4efb4834c074c13a954b83b14ddf289 P 6e489a4674b840c4b61f170102698b7d: Deleting WAL directory at /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713339161492-17621-0/minicluster-data/ts-0/wal/wals/a4efb4834c074c13a954b83b14ddf289
I20260929 20:27:02.943610 1913 ts_tablet_manager.cc:1939] T a4efb4834c074c13a954b83b14ddf289 P 6e489a4674b840c4b61f170102698b7d: Deleting consensus metadata
I20260929 20:27:02.943934 1850 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260929 20:27:02.944079 1850 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.015s user 0.001s sys 0.000s
I20260929 20:27:02.944219 1850 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:27:02.944307 1850 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:27:02.980525 1850 rpc_server.cc:307] RPC server started. Bound to: 127.17.53.65:43287
I20260929 20:27:02.980614 2019 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.53.65:43287 every 8 connection(s)
I20260929 20:27:02.981913 1850 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713339161492-17621-0/minicluster-data/ts-0/data/info.pb
I20260929 20:27:02.986676 17621 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu as pid 1850
I20260929 20:27:02.987673 17621 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
/tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713339161492-17621-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713339161492-17621-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713339161492-17621-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713339161492-17621-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.53.66:33821
--local_ip_for_outbound_sockets=127.17.53.66
--tserver_master_addrs=127.17.53.126:39625
--webserver_port=37051
--webserver_interface=127.17.53.66
--keytab_file=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.66.keytab
--principal=kudu/127.17.53.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.53.84:36709
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713339161492-17621-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713339161492-17621-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713339161492-17621-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 20:27:02.987761 2022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:27:02.985191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:39625 (local address 127.17.53.65:46849)
0929 20:27:02.986025 (+ 834us) negotiation.cc:107] Waiting for socket to connect
0929 20:27:02.986364 (+ 339us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.53.126:39625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":513,"thread_start_us":301,"threads_started":1}
W20260929 20:27:02.988605 2020 heartbeater.cc:644] Failed to heartbeat to 127.17.53.126:39625 (0 consecutive failures): Network error: Failed to ping master at 127.17.53.126:39625: Client connection negotiation failed: client connection to 127.17.53.126:39625: connect: Connection refused (error 111)
I20260929 20:27:02.989595 2022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:27:02.989257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:39625 (local address 127.17.53.65:51705)
0929 20:27:02.989422 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0929 20:27:02.989494 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.53.126:39625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260929 20:27:02.990558 2022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:27:02.990200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:39625 (local address 127.17.53.65:43485)
0929 20:27:02.990348 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0929 20:27:02.990445 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.53.126:39625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260929 20:27:02.990833 2020 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 20:27:03.329676 2023 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:27:03.330037 2023 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:27:03.330228 2023 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:27:03.338450 2023 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:27:03.338622 2023 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:27:03.338824 2023 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.53.66
I20260929 20:27:03.348847 2023 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.53.84:36709
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713339161492-17621-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713339161492-17621-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.66.keytab
--principal=kudu/127.17.53.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.53.66:33821
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713339161492-17621-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.17.53.66
--webserver_port=37051
--tserver_master_addrs=127.17.53.126:39625
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.53.66
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713339161492-17621-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:27:03.350238 2023 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:27:03.351670 2023 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:27:03.365429 2031 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:27:03.365374 2029 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:27:03.366011 2028 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:27:03.366466 2023 server_base.cc:1034] running on GCE node
I20260929 20:27:03.367163 2023 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:27:03.368162 2023 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:27:03.369416 2023 hybrid_clock.cc:648] HybridClock initialized: now 1790713623369368 us; error 62 us; skew 500 ppm
I20260929 20:27:03.369748 2023 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:27:03 dist-test-slave-zbnc krb5kdc[699](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713623, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.53.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:27:03.373258 2023 init.cc:377] Logged in from keytab as kudu/127.17.53.66@KRBTEST.COM (short username kudu)
I20260929 20:27:03.375027 2023 webserver.cc:460] Webserver started at http://127.17.53.66:37051/ using document root <none> and password file <none>
I20260929 20:27:03.375792 2023 fs_manager.cc:359] Metadata directory not provided
I20260929 20:27:03.375972 2023 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:27:03.382133 2023 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.004s
I20260929 20:27:03.385123 2038 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:27:03.386806 2023 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 20:27:03.387085 2023 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713339161492-17621-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713339161492-17621-0/minicluster-data/ts-1/wal
uuid: "3b59fcbacafc43b088c6cacc664f6ccf"
format_stamp: "Formatted at 2026-09-29 20:26:17 on dist-test-slave-zbnc"
server_key: "79ab5d60f09934e4ae2ad1cda3c47a82"
server_key_iv: "a2995c358f36bfdcdfa0e549a65cbd2e"
server_key_version: "encryptionkey@0"
I20260929 20:27:03.388312 2023 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713339161492-17621-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713339161492-17621-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713339161492-17621-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:27:03.413532 2023 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:27:03.418547 2023 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:27:03.418917 2023 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:27:03.420049 2023 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:27:03.421825 2023 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:27:03.421975 2023 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:27:03.422125 2023 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:27:03.422227 2023 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:27:03.461202 2023 rpc_server.cc:307] RPC server started. Bound to: 127.17.53.66:33821
I20260929 20:27:03.461285 2150 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.53.66:33821 every 8 connection(s)
I20260929 20:27:03.462786 2023 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713339161492-17621-0/minicluster-data/ts-1/data/info.pb
I20260929 20:27:03.463656 17621 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu as pid 2023
I20260929 20:27:03.464558 17621 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
/tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713339161492-17621-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713339161492-17621-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713339161492-17621-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713339161492-17621-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.53.67:45491
--local_ip_for_outbound_sockets=127.17.53.67
--tserver_master_addrs=127.17.53.126:39625
--webserver_port=41667
--webserver_interface=127.17.53.67
--keytab_file=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.67.keytab
--principal=kudu/127.17.53.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.53.84:36709
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713339161492-17621-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713339161492-17621-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713339161492-17621-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 20:27:03.468269 2153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:27:03.466112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:39625 (local address 127.17.53.66:51793)
0929 20:27:03.466984 (+ 872us) negotiation.cc:107] Waiting for socket to connect
0929 20:27:03.467313 (+ 329us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.53.126:39625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":555,"thread_start_us":365,"threads_started":1}
W20260929 20:27:03.469096 2151 heartbeater.cc:644] Failed to heartbeat to 127.17.53.126:39625 (0 consecutive failures): Network error: Failed to ping master at 127.17.53.126:39625: Client connection negotiation failed: client connection to 127.17.53.126:39625: connect: Connection refused (error 111)
I20260929 20:27:03.469993 2153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:27:03.469643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:39625 (local address 127.17.53.66:53373)
0929 20:27:03.469802 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0929 20:27:03.469885 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.53.126:39625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260929 20:27:03.470969 2153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:27:03.470580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:39625 (local address 127.17.53.66:42553)
0929 20:27:03.470722 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0929 20:27:03.470805 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.53.126:39625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260929 20:27:03.471280 2151 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260929 20:27:03.795670 2154 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:27:03.795959 2154 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:27:03.796059 2154 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:27:03.803287 2154 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:27:03.803401 2154 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:27:03.803514 2154 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.53.67
I20260929 20:27:03.813248 2154 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.53.84:36709
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713339161492-17621-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713339161492-17621-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.67.keytab
--principal=kudu/127.17.53.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.53.67:45491
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713339161492-17621-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.17.53.67
--webserver_port=41667
--tserver_master_addrs=127.17.53.126:39625
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.53.67
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713339161492-17621-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:27:03.814531 2154 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:27:03.815817 2154 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:27:03.829003 2159 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:27:03.829084 2160 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:27:03.829208 2162 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:27:03.829855 2154 server_base.cc:1034] running on GCE node
I20260929 20:27:03.830416 2154 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:27:03.831393 2154 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:27:03.832592 2154 hybrid_clock.cc:648] HybridClock initialized: now 1790713623832544 us; error 39 us; skew 500 ppm
I20260929 20:27:03.832868 2154 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:27:03 dist-test-slave-zbnc krb5kdc[699](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713623, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.53.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:27:03.836115 2154 init.cc:377] Logged in from keytab as kudu/127.17.53.67@KRBTEST.COM (short username kudu)
I20260929 20:27:03.837635 2154 webserver.cc:460] Webserver started at http://127.17.53.67:41667/ using document root <none> and password file <none>
I20260929 20:27:03.838310 2154 fs_manager.cc:359] Metadata directory not provided
I20260929 20:27:03.838490 2154 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:27:03.844174 2154 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260929 20:27:03.847208 2169 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:27:03.848630 2154 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 20:27:03.848826 2154 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713339161492-17621-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713339161492-17621-0/minicluster-data/ts-2/wal
uuid: "445d9c75756f4730a2cb5bb98e8814e9"
format_stamp: "Formatted at 2026-09-29 20:26:18 on dist-test-slave-zbnc"
server_key: "730b50875bdd2ab9fe8dd6c354970414"
server_key_iv: "a91a7eb8fc8fda82009f81892050fd7d"
server_key_version: "encryptionkey@0"
I20260929 20:27:03.849927 2154 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713339161492-17621-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713339161492-17621-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713339161492-17621-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:27:03.868479 2154 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:27:03.873082 2154 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:27:03.873363 2154 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:27:03.874402 2154 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:27:03.875878 2154 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:27:03.876006 2154 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:27:03.876135 2154 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:27:03.876224 2154 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:27:03.911285 2154 rpc_server.cc:307] RPC server started. Bound to: 127.17.53.67:45491
I20260929 20:27:03.911382 2282 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.53.67:45491 every 8 connection(s)
I20260929 20:27:03.913256 2154 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790713339161492-17621-0/minicluster-data/ts-2/data/info.pb
I20260929 20:27:03.918530 2285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:27:03.916511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:39625 (local address 127.17.53.67:37513)
0929 20:27:03.917345 (+ 834us) negotiation.cc:107] Waiting for socket to connect
0929 20:27:03.917572 (+ 227us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.53.126:39625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":559,"thread_start_us":356,"threads_started":1}
W20260929 20:27:03.919358 2283 heartbeater.cc:644] Failed to heartbeat to 127.17.53.126:39625 (0 consecutive failures): Network error: Failed to ping master at 127.17.53.126:39625: Client connection negotiation failed: client connection to 127.17.53.126:39625: connect: Connection refused (error 111)
I20260929 20:27:03.919466 17621 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu as pid 2154
I20260929 20:27:03.920604 2285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:27:03.919998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:39625 (local address 127.17.53.67:51471)
0929 20:27:03.920326 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0929 20:27:03.920444 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.53.126:39625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260929 20:27:03.921873 2285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:27:03.921539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:39625 (local address 127.17.53.67:59805)
0929 20:27:03.921680 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0929 20:27:03.921757 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.53.126:39625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260929 20:27:03.922159 2283 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260929 20:27:03.992429 2287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:27:03.991562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:39625 (local address 127.17.53.65:60041)
0929 20:27:03.992080 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0929 20:27:03.992182 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.53.126:39625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":406,"thread_start_us":276,"threads_started":1}
I20260929 20:27:04.472781 2288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:27:04.471830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:39625 (local address 127.17.53.66:58613)
0929 20:27:04.472443 (+ 613us) negotiation.cc:107] Waiting for socket to connect
0929 20:27:04.472537 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.53.126:39625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":491,"thread_start_us":357,"threads_started":1}
I20260929 20:27:04.923910 2289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:27:04.922784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:39625 (local address 127.17.53.67:46423)
0929 20:27:04.923563 (+ 779us) negotiation.cc:107] Waiting for socket to connect
0929 20:27:04.923676 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.53.126:39625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":628,"thread_start_us":506,"threads_started":1}
I20260929 20:27:04.994266 2290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:27:04.993534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:39625 (local address 127.17.53.65:35523)
0929 20:27:04.994027 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0929 20:27:04.994111 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.53.126:39625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":224,"threads_started":1}
I20260929 20:27:05.474836 2291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:27:05.473839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:39625 (local address 127.17.53.66:52521)
0929 20:27:05.474518 (+ 679us) negotiation.cc:107] Waiting for socket to connect
0929 20:27:05.474621 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.53.126:39625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":540,"thread_start_us":366,"threads_started":1}
I20260929 20:27:05.925895 2292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:27:05.924992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:39625 (local address 127.17.53.67:38443)
0929 20:27:05.925602 (+ 610us) negotiation.cc:107] Waiting for socket to connect
0929 20:27:05.925699 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.53.126:39625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":492,"thread_start_us":331,"threads_started":1}
I20260929 20:27:05.996112 2293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:27:05.995317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:39625 (local address 127.17.53.65:50441)
0929 20:27:05.995825 (+ 508us) negotiation.cc:107] Waiting for socket to connect
0929 20:27:05.995921 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.53.126:39625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":401,"thread_start_us":263,"threads_started":1}
I20260929 20:27:06.476877 2294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:27:06.475997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:39625 (local address 127.17.53.66:46675)
0929 20:27:06.476553 (+ 556us) negotiation.cc:107] Waiting for socket to connect
0929 20:27:06.476667 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.53.126:39625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":430,"thread_start_us":293,"threads_started":1}
I20260929 20:27:06.927917 2295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:27:06.927075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:39625 (local address 127.17.53.67:57905)
0929 20:27:06.927609 (+ 534us) negotiation.cc:107] Waiting for socket to connect
0929 20:27:06.927691 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.53.126:39625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":292,"threads_started":1}
I20260929 20:27:06.997819 2296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:27:06.997055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:39625 (local address 127.17.53.65:40717)
0929 20:27:06.997552 (+ 497us) negotiation.cc:107] Waiting for socket to connect
0929 20:27:06.997645 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.53.126:39625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":387,"thread_start_us":243,"threads_started":1}
I20260929 20:27:07.478729 2297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:27:07.477936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:39625 (local address 127.17.53.66:35569)
0929 20:27:07.478454 (+ 518us) negotiation.cc:107] Waiting for socket to connect
0929 20:27:07.478554 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.53.126:39625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":402,"thread_start_us":270,"threads_started":1}
I20260929 20:27:07.929559 2298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:27:07.928826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:39625 (local address 127.17.53.67:60271)
0929 20:27:07.929274 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0929 20:27:07.929362 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.53.126:39625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":338,"thread_start_us":217,"threads_started":1}
I20260929 20:27:07.999760 2299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:27:07.999011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:39625 (local address 127.17.53.65:57275)
0929 20:27:07.999496 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0929 20:27:07.999584 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.53.126:39625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":242,"threads_started":1}
I20260929 20:27:08.480494 2300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:27:08.479667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:39625 (local address 127.17.53.66:60921)
0929 20:27:08.480228 (+ 561us) negotiation.cc:107] Waiting for socket to connect
0929 20:27:08.480320 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.53.126:39625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":444,"thread_start_us":314,"threads_started":1}
I20260929 20:27:08.931489 2301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:27:08.930597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:39625 (local address 127.17.53.67:52403)
0929 20:27:08.931209 (+ 612us) negotiation.cc:107] Waiting for socket to connect
0929 20:27:08.931295 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.53.126:39625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":505,"thread_start_us":389,"threads_started":1}
I20260929 20:27:09.001626 2302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:27:09.000762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:39625 (local address 127.17.53.65:51063)
0929 20:27:09.001374 (+ 612us) negotiation.cc:107] Waiting for socket to connect
0929 20:27:09.001462 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.53.126:39625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":493,"thread_start_us":366,"threads_started":1}
I20260929 20:27:09.482203 2303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:27:09.481435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:39625 (local address 127.17.53.66:37687)
0929 20:27:09.481919 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0929 20:27:09.482015 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.53.126:39625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":250,"threads_started":1}
I20260929 20:27:09.933212 2304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:27:09.932412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:39625 (local address 127.17.53.67:47935)
0929 20:27:09.932900 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0929 20:27:09.933048 (+ 148us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.53.126:39625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":388,"thread_start_us":266,"threads_started":1}
I20260929 20:27:10.003453 2305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:27:10.002483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:39625 (local address 127.17.53.65:39597)
0929 20:27:10.003147 (+ 664us) negotiation.cc:107] Waiting for socket to connect
0929 20:27:10.003233 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.53.126:39625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":551,"thread_start_us":243,"threads_started":1}
I20260929 20:27:10.484184 2306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:27:10.483314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:39625 (local address 127.17.53.66:53975)
0929 20:27:10.483862 (+ 548us) negotiation.cc:107] Waiting for socket to connect
0929 20:27:10.483965 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.53.126:39625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":434,"thread_start_us":262,"threads_started":1}
I20260929 20:27:10.935045 2307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:27:10.934174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:39625 (local address 127.17.53.67:55571)
0929 20:27:10.934673 (+ 499us) negotiation.cc:107] Waiting for socket to connect
0929 20:27:10.934756 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.53.126:39625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":237,"threads_started":1}
I20260929 20:27:11.005255 2308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:27:11.004440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:39625 (local address 127.17.53.65:51895)
0929 20:27:11.005008 (+ 568us) negotiation.cc:107] Waiting for socket to connect
0929 20:27:11.005095 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.53.126:39625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":341,"threads_started":1}
I20260929 20:27:11.486270 2309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:27:11.485239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:39625 (local address 127.17.53.66:44987)
0929 20:27:11.485934 (+ 695us) negotiation.cc:107] Waiting for socket to connect
0929 20:27:11.486050 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.53.126:39625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":485,"thread_start_us":332,"threads_started":1}
I20260929 20:27:11.936939 2310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:27:11.936216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:39625 (local address 127.17.53.67:47875)
0929 20:27:11.936681 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0929 20:27:11.936766 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.53.126:39625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":370,"thread_start_us":258,"threads_started":1}
I20260929 20:27:12.007095 2311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:27:12.006188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:39625 (local address 127.17.53.65:42383)
0929 20:27:12.006683 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0929 20:27:12.006787 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.53.126:39625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":382,"thread_start_us":252,"threads_started":1}
I20260929 20:27:12.488447 2312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:27:12.487579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:39625 (local address 127.17.53.66:49619)
0929 20:27:12.488182 (+ 603us) negotiation.cc:107] Waiting for socket to connect
0929 20:27:12.488265 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.53.126:39625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":489,"thread_start_us":306,"threads_started":1}
I20260929 20:27:12.938665 2313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:27:12.937942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:39625 (local address 127.17.53.67:50209)
0929 20:27:12.938401 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0929 20:27:12.938489 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.53.126:39625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":357,"thread_start_us":223,"threads_started":1}
I20260929 20:27:13.008783 2314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:27:13.008049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:39625 (local address 127.17.53.65:57441)
0929 20:27:13.008530 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0929 20:27:13.008618 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.53.126:39625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":242,"threads_started":1}
I20260929 20:27:13.490479 2315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:27:13.489591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:39625 (local address 127.17.53.66:36303)
0929 20:27:13.490138 (+ 547us) negotiation.cc:107] Waiting for socket to connect
0929 20:27:13.490244 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.53.126:39625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":423,"thread_start_us":320,"threads_started":1}
I20260929 20:27:13.940624 2316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:27:13.939758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:39625 (local address 127.17.53.67:54171)
0929 20:27:13.940353 (+ 595us) negotiation.cc:107] Waiting for socket to connect
0929 20:27:13.940448 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.53.126:39625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":351,"threads_started":1}
I20260929 20:27:14.010731 2317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:27:14.009809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:39625 (local address 127.17.53.65:42093)
0929 20:27:14.010433 (+ 624us) negotiation.cc:107] Waiting for socket to connect
0929 20:27:14.010527 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.53.126:39625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":467,"thread_start_us":337,"threads_started":1}
I20260929 20:27:14.492527 2318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:27:14.491507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:39625 (local address 127.17.53.66:53573)
0929 20:27:14.492139 (+ 632us) negotiation.cc:107] Waiting for socket to connect
0929 20:27:14.492274 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.53.126:39625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":485,"thread_start_us":320,"threads_started":1}
I20260929 20:27:14.942477 2319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:27:14.941714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:39625 (local address 127.17.53.67:53123)
0929 20:27:14.942200 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0929 20:27:14.942294 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.53.126:39625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":387,"thread_start_us":266,"threads_started":1}
I20260929 20:27:15.012642 2320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:27:15.011775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:39625 (local address 127.17.53.65:42915)
0929 20:27:15.012354 (+ 579us) negotiation.cc:107] Waiting for socket to connect
0929 20:27:15.012453 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.53.126:39625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":444,"thread_start_us":321,"threads_started":1}
I20260929 20:27:15.494400 2321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:27:15.493567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:39625 (local address 127.17.53.66:40245)
0929 20:27:15.494127 (+ 560us) negotiation.cc:107] Waiting for socket to connect
0929 20:27:15.494211 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.53.126:39625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":416,"thread_start_us":332,"threads_started":1}
I20260929 20:27:15.944375 2322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:27:15.943608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:39625 (local address 127.17.53.67:54471)
0929 20:27:15.944091 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0929 20:27:15.944183 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.53.126:39625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":359,"thread_start_us":255,"threads_started":1}
I20260929 20:27:16.014636 2323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:27:16.013789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:39625 (local address 127.17.53.65:46345)
0929 20:27:16.014338 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0929 20:27:16.014442 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.53.126:39625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":443,"thread_start_us":296,"threads_started":1}
I20260929 20:27:16.496313 2324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:27:16.495420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:39625 (local address 127.17.53.66:49225)
0929 20:27:16.496012 (+ 592us) negotiation.cc:107] Waiting for socket to connect
0929 20:27:16.496114 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.53.126:39625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":490,"thread_start_us":311,"threads_started":1}
I20260929 20:27:16.946357 2325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:27:16.945422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:39625 (local address 127.17.53.67:39899)
0929 20:27:16.946061 (+ 639us) negotiation.cc:107] Waiting for socket to connect
0929 20:27:16.946171 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.53.126:39625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":502,"thread_start_us":336,"threads_started":1}
I20260929 20:27:17.016285 2326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:27:17.015569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:39625 (local address 127.17.53.65:47041)
0929 20:27:17.016020 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0929 20:27:17.016109 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.53.126:39625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":218,"threads_started":1}
I20260929 20:27:17.498080 2327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:27:17.497258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:39625 (local address 127.17.53.66:37897)
0929 20:27:17.497737 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0929 20:27:17.497826 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.53.126:39625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":380,"thread_start_us":270,"threads_started":1}
I20260929 20:27:17.948027 2328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:27:17.947268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:39625 (local address 127.17.53.67:52681)
0929 20:27:17.947701 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0929 20:27:17.947782 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.53.126:39625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":337,"thread_start_us":231,"threads_started":1}
I20260929 20:27:18.018292 2329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:27:18.017477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:39625 (local address 127.17.53.65:40195)
0929 20:27:18.018004 (+ 527us) negotiation.cc:107] Waiting for socket to connect
0929 20:27:18.018113 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.53.126:39625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":242,"threads_started":1}
I20260929 20:27:18.500036 2330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:27:18.499195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:39625 (local address 127.17.53.66:54725)
0929 20:27:18.499687 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0929 20:27:18.499826 (+ 139us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.53.126:39625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":368,"thread_start_us":248,"threads_started":1}
I20260929 20:27:18.924186 17621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu pid 1850
I20260929 20:27:18.950223 2332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:27:18.949296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:39625 (local address 127.17.53.67:57199)
0929 20:27:18.949858 (+ 562us) negotiation.cc:107] Waiting for socket to connect
0929 20:27:18.949989 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.53.126:39625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":408,"thread_start_us":274,"threads_started":1}
I20260929 20:27:18.952597 2331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:27:18.925803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45588 (local address 127.17.53.65:43287)
0929 20:27:18.926522 (+ 719us) server_negotiation.cc:206] Beginning negotiation
0929 20:27:18.926548 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0929 20:27:18.926608 (+ 60us) server_negotiation.cc:407] Connection header received
0929 20:27:18.928647 (+ 2039us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:27:18.928659 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:27:18.929155 (+ 496us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:27:18.929499 (+ 344us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:27:18.931299 (+ 1800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:27:18.932821 (+ 1522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:27:18.934354 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:27:18.935273 (+ 919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:27:18.935440 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:27:18.944945 (+ 9505us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:27:18.944956 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:27:18.944978 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:27:18.945036 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:27:18.948066 (+ 3030us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:27:18.949200 (+ 1134us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:27:18.949209 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:27:18.949218 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:27:18.949441 (+ 223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:27:18.950071 (+ 630us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:27:18.950075 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:27:18.950077 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:27:18.950731 (+ 654us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:27:18.951028 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0929 20:27:18.951623 (+ 595us) server_negotiation.cc:299] Negotiation successful
0929 20:27:18.952134 (+ 511us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":460,"thread_start_us":325,"threads_started":1}
I20260929 20:27:18.954124 2015 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:27:19.044900 2334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:27:19.043667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:39625 (local address 127.17.53.65:39673)
0929 20:27:19.044317 (+ 650us) negotiation.cc:107] Waiting for socket to connect
0929 20:27:19.044567 (+ 250us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.53.126:39625: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":481,"thread_start_us":344,"threads_started":1}
I20260929 20:27:19.044914 17621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu with pid 1850
I20260929 20:27:19.064996 17621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu pid 2023
I20260929 20:27:19.084502 2335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:27:19.065983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39848 (local address 127.17.53.66:33821)
0929 20:27:19.066522 (+ 539us) server_negotiation.cc:206] Beginning negotiation
0929 20:27:19.066536 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 20:27:19.066574 (+ 38us) server_negotiation.cc:407] Connection header received
0929 20:27:19.068372 (+ 1798us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:27:19.068383 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:27:19.068827 (+ 444us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:27:19.069190 (+ 363us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:27:19.070718 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:27:19.072062 (+ 1344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:27:19.073359 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:27:19.074060 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:27:19.074269 (+ 209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:27:19.077484 (+ 3215us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:27:19.077492 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:27:19.077515 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:27:19.077557 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:27:19.080851 (+ 3294us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:27:19.081624 (+ 773us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:27:19.081632 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:27:19.081640 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:27:19.081949 (+ 309us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:27:19.082392 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:27:19.082396 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:27:19.082398 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:27:19.083134 (+ 736us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:27:19.083381 (+ 247us) server_negotiation.cc:1035] Waiting for connection context
0929 20:27:19.083785 (+ 404us) server_negotiation.cc:299] Negotiation successful
0929 20:27:19.084211 (+ 426us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":387,"thread_start_us":249,"threads_started":1}
I20260929 20:27:19.086200 2146 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:27:19.171147 17621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu with pid 2023
I20260929 20:27:19.189010 17621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu pid 2154
I20260929 20:27:19.208065 2337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:27:19.190137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52858 (local address 127.17.53.67:45491)
0929 20:27:19.190793 (+ 656us) server_negotiation.cc:206] Beginning negotiation
0929 20:27:19.190807 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 20:27:19.190848 (+ 41us) server_negotiation.cc:407] Connection header received
0929 20:27:19.192697 (+ 1849us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:27:19.192707 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:27:19.193134 (+ 427us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:27:19.193498 (+ 364us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:27:19.195026 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:27:19.196238 (+ 1212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:27:19.197700 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:27:19.198387 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:27:19.198595 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:27:19.201538 (+ 2943us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:27:19.201546 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:27:19.201562 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:27:19.201605 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:27:19.204483 (+ 2878us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:27:19.205219 (+ 736us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:27:19.205225 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:27:19.205233 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:27:19.205476 (+ 243us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:27:19.206003 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:27:19.206008 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:27:19.206012 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:27:19.206601 (+ 589us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:27:19.206844 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0929 20:27:19.207298 (+ 454us) server_negotiation.cc:299] Negotiation successful
0929 20:27:19.207753 (+ 455us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":496,"thread_start_us":331,"threads_started":1}
I20260929 20:27:19.209455 2278 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:27:19.296356 17621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu with pid 2154
2026-09-29T20:27:19Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (64494 ms)
[----------] 30 tests from SecurityITest (300098 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790713339161492-17621-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 20:27:19 dist-test-slave-zbnc krb5kdc[2341](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 20:27:19 dist-test-slave-zbnc krb5kdc[2341](info): set up 2 sockets
Sep 29 20:27:19 dist-test-slave-zbnc krb5kdc[2341](info): commencing operation
krb5kdc: starting...
W20260929 20:27:20.436493 17621 mini_kdc.cc:121] Time spent starting KDC: real 1.100s user 0.007s sys 0.019s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790713339161492-17621-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790713339161492-17621-0/krb5kdc/test-user.keytab.
Sep 29 20:27:20 dist-test-slave-zbnc krb5kdc[2341](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713640, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T20:27:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T20:27:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.17.53.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.53.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.53.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.126.keytab.
Entry for principal kudu/127.17.53.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.126.keytab.
I20260929 20:27:20.691073 17621 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
/tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790713339161492-17621-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790713339161492-17621-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790713339161492-17621-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790713339161492-17621-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.53.126:38017
--webserver_interface=127.17.53.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.126.keytab
--principal=kudu/127.17.53.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.53.84:33833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.53.126:38017
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790713339161492-17621-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790713339161492-17621-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790713339161492-17621-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:27:21.028528 2354 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:27:21.028825 2354 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:27:21.028990 2354 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:27:21.036398 2354 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:27:21.036511 2354 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:27:21.036607 2354 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:27:21.036706 2354 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:27:21.036789 2354 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:27:21.046988 2354 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.53.84:33833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790713339161492-17621-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790713339161492-17621-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.53.126:38017
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.126.keytab
--principal=kudu/127.17.53.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.53.126:38017
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790713339161492-17621-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.53.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790713339161492-17621-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:27:21.048389 2354 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:27:21.050169 2354 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:27:21.065367 2359 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:27:21.065667 2360 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:27:21.065478 2362 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:27:21.066458 2354 server_base.cc:1034] running on GCE node
I20260929 20:27:21.067322 2354 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:27:21.068768 2354 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:27:21.070020 2354 hybrid_clock.cc:648] HybridClock initialized: now 1790713641069973 us; error 28 us; skew 500 ppm
I20260929 20:27:21.070396 2354 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:27:21 dist-test-slave-zbnc krb5kdc[2341](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713641, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.53.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:27:21.073719 2354 init.cc:377] Logged in from keytab as kudu/127.17.53.126@KRBTEST.COM (short username kudu)
I20260929 20:27:21.075441 2354 webserver.cc:460] Webserver started at http://127.17.53.126:35689/ using document root <none> and password file <none>
I20260929 20:27:21.076179 2354 fs_manager.cc:359] Metadata directory not provided
I20260929 20:27:21.076326 2354 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:27:21.076646 2354 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:27:21.079706 2354 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790713339161492-17621-0/minicluster-data/master-0/data/instance:
uuid: "58314c300c934d2bbef023f492f2cafe"
format_stamp: "Formatted at 2026-09-29 20:27:21 on dist-test-slave-zbnc"
server_key: "16a682a18a5893c8b4216d66e495a707"
server_key_iv: "858a15f9c5ccffe2b20f3b0d95788d93"
server_key_version: "encryptionkey@0"
I20260929 20:27:21.080516 2354 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790713339161492-17621-0/minicluster-data/master-0/wal/instance:
uuid: "58314c300c934d2bbef023f492f2cafe"
format_stamp: "Formatted at 2026-09-29 20:27:21 on dist-test-slave-zbnc"
server_key: "16a682a18a5893c8b4216d66e495a707"
server_key_iv: "858a15f9c5ccffe2b20f3b0d95788d93"
server_key_version: "encryptionkey@0"
I20260929 20:27:21.085602 2354 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260929 20:27:21.089031 2369 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:27:21.090405 2354 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260929 20:27:21.090588 2354 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790713339161492-17621-0/minicluster-data/master-0/data,/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790713339161492-17621-0/minicluster-data/master-0/wal
uuid: "58314c300c934d2bbef023f492f2cafe"
format_stamp: "Formatted at 2026-09-29 20:27:21 on dist-test-slave-zbnc"
server_key: "16a682a18a5893c8b4216d66e495a707"
server_key_iv: "858a15f9c5ccffe2b20f3b0d95788d93"
server_key_version: "encryptionkey@0"
I20260929 20:27:21.090780 2354 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790713339161492-17621-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790713339161492-17621-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790713339161492-17621-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:27:21.111501 2354 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:27:21.115877 2354 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:27:21.116182 2354 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:27:21.136142 2354 rpc_server.cc:307] RPC server started. Bound to: 127.17.53.126:38017
I20260929 20:27:21.136183 2420 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.53.126:38017 every 8 connection(s)
I20260929 20:27:21.137573 2354 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790713339161492-17621-0/minicluster-data/master-0/data/info.pb
I20260929 20:27:21.141661 2421 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:27:21.145982 17621 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu as pid 2354
I20260929 20:27:21.146242 17621 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790713339161492-17621-0/minicluster-data/master-0/wal/instance
I20260929 20:27:21.147212 17621 external_mini_cluster.cc:1428] Setting key 3c8ca88ba072b9e29e0b474ccebf8d2d
I20260929 20:27:21.155922 2421 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 58314c300c934d2bbef023f492f2cafe: Bootstrap starting.
I20260929 20:27:21.159379 2421 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 58314c300c934d2bbef023f492f2cafe: Neither blocks nor log segments found. Creating new log.
I20260929 20:27:21.160452 2421 log.cc:824] T 00000000000000000000000000000000 P 58314c300c934d2bbef023f492f2cafe: Log is configured to *not* fsync() on all Append() calls
Sep 29 20:27:21 dist-test-slave-zbnc krb5kdc[2341](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713640, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.53.126@KRBTEST.COM
I20260929 20:27:21.163746 2421 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 58314c300c934d2bbef023f492f2cafe: No bootstrap required, opened a new log
I20260929 20:27:21.169792 2424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:27:21.149547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50000 (local address 127.17.53.126:38017)
0929 20:27:21.150309 (+ 762us) server_negotiation.cc:206] Beginning negotiation
0929 20:27:21.150321 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 20:27:21.150360 (+ 39us) server_negotiation.cc:407] Connection header received
0929 20:27:21.152096 (+ 1736us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:27:21.152118 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:27:21.152530 (+ 412us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:27:21.152944 (+ 414us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:27:21.154433 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:27:21.155797 (+ 1364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:27:21.157195 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:27:21.157899 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:27:21.158090 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:27:21.162442 (+ 4352us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:27:21.162450 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:27:21.162465 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:27:21.162499 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:27:21.165740 (+ 3241us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:27:21.166438 (+ 698us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:27:21.166444 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:27:21.166449 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:27:21.166612 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:27:21.167081 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:27:21.167092 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:27:21.167095 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:27:21.167602 (+ 507us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:27:21.167898 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0929 20:27:21.168374 (+ 476us) server_negotiation.cc:299] Negotiation successful
0929 20:27:21.168806 (+ 432us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":554,"thread_start_us":333,"threads_started":1}
I20260929 20:27:21.172717 2421 raft_consensus.cc:348] T 00000000000000000000000000000000 P 58314c300c934d2bbef023f492f2cafe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58314c300c934d2bbef023f492f2cafe" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 38017 } }
I20260929 20:27:21.173163 2421 raft_consensus.cc:374] T 00000000000000000000000000000000 P 58314c300c934d2bbef023f492f2cafe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:27:21.173292 2421 raft_consensus.cc:729] T 00000000000000000000000000000000 P 58314c300c934d2bbef023f492f2cafe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58314c300c934d2bbef023f492f2cafe, State: Initialized, Role: FOLLOWER
I20260929 20:27:21.173933 2421 consensus_queue.cc:260] T 00000000000000000000000000000000 P 58314c300c934d2bbef023f492f2cafe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58314c300c934d2bbef023f492f2cafe" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 38017 } }
I20260929 20:27:21.174227 2421 raft_consensus.cc:388] T 00000000000000000000000000000000 P 58314c300c934d2bbef023f492f2cafe [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 20:27:21.174420 2421 raft_consensus.cc:482] T 00000000000000000000000000000000 P 58314c300c934d2bbef023f492f2cafe [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 20:27:21.174626 2421 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 58314c300c934d2bbef023f492f2cafe [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:27:21.177027 2421 raft_consensus.cc:504] T 00000000000000000000000000000000 P 58314c300c934d2bbef023f492f2cafe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58314c300c934d2bbef023f492f2cafe" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 38017 } }
I20260929 20:27:21.177546 2421 leader_election.cc:302] T 00000000000000000000000000000000 P 58314c300c934d2bbef023f492f2cafe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 58314c300c934d2bbef023f492f2cafe; no voters:
I20260929 20:27:21.178242 2421 leader_election.cc:288] T 00000000000000000000000000000000 P 58314c300c934d2bbef023f492f2cafe [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 20:27:21.178380 2426 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 58314c300c934d2bbef023f492f2cafe [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:27:21.179162 2426 raft_consensus.cc:686] T 00000000000000000000000000000000 P 58314c300c934d2bbef023f492f2cafe [term 1 LEADER]: Becoming Leader. State: Replica: 58314c300c934d2bbef023f492f2cafe, State: Running, Role: LEADER
I20260929 20:27:21.179811 2426 consensus_queue.cc:237] T 00000000000000000000000000000000 P 58314c300c934d2bbef023f492f2cafe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58314c300c934d2bbef023f492f2cafe" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 38017 } }
I20260929 20:27:21.180615 2421 sys_catalog.cc:564] T 00000000000000000000000000000000 P 58314c300c934d2bbef023f492f2cafe [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:27:21.182809 2427 sys_catalog.cc:455] T 00000000000000000000000000000000 P 58314c300c934d2bbef023f492f2cafe [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "58314c300c934d2bbef023f492f2cafe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58314c300c934d2bbef023f492f2cafe" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 38017 } } }
I20260929 20:27:21.182964 2428 sys_catalog.cc:455] T 00000000000000000000000000000000 P 58314c300c934d2bbef023f492f2cafe [sys.catalog]: SysCatalogTable state changed. Reason: New leader 58314c300c934d2bbef023f492f2cafe. Latest consensus state: current_term: 1 leader_uuid: "58314c300c934d2bbef023f492f2cafe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "58314c300c934d2bbef023f492f2cafe" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 38017 } } }
I20260929 20:27:21.183645 2428 sys_catalog.cc:458] T 00000000000000000000000000000000 P 58314c300c934d2bbef023f492f2cafe [sys.catalog]: This master's current role is: LEADER
I20260929 20:27:21.183809 2427 sys_catalog.cc:458] T 00000000000000000000000000000000 P 58314c300c934d2bbef023f492f2cafe [sys.catalog]: This master's current role is: LEADER
I20260929 20:27:21.185808 2433 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:27:21.191762 2433 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:27:21.201853 2433 catalog_manager.cc:1347] Generated new cluster ID: 2b68108561fb48a5a4d9850fdea23b3b
I20260929 20:27:21.202104 2433 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:27:21.213356 2433 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 20:27:21.214732 2433 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:27:21.229065 2433 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 58314c300c934d2bbef023f492f2cafe: Generated new TSK 0
I20260929 20:27:21.229959 2433 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.17.53.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.53.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.53.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.65.keytab.
Entry for principal kudu/127.17.53.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.65.keytab.
I20260929 20:27:21.299335 17621 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
/tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790713339161492-17621-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790713339161492-17621-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790713339161492-17621-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790713339161492-17621-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.53.65:0
--local_ip_for_outbound_sockets=127.17.53.65
--webserver_interface=127.17.53.65
--webserver_port=0
--tserver_master_addrs=127.17.53.126:38017
--keytab_file=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.65.keytab
--principal=kudu/127.17.53.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.53.84:33833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790713339161492-17621-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790713339161492-17621-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790713339161492-17621-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:27:21.658468 2447 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:27:21.658780 2447 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:27:21.658936 2447 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:27:21.666556 2447 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:27:21.666672 2447 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:27:21.666785 2447 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.53.65
I20260929 20:27:21.676815 2447 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.53.84:33833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790713339161492-17621-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790713339161492-17621-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.65.keytab
--principal=kudu/127.17.53.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.53.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790713339161492-17621-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.17.53.65
--webserver_port=0
--tserver_master_addrs=127.17.53.126:38017
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.53.65
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790713339161492-17621-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:27:21.678048 2447 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:27:21.679461 2447 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:27:21.693508 2455 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:27:21.693665 2453 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:27:21.693663 2452 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:27:21.694820 2447 server_base.cc:1034] running on GCE node
I20260929 20:27:21.695611 2447 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:27:21.696636 2447 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:27:21.697944 2447 hybrid_clock.cc:648] HybridClock initialized: now 1790713641697900 us; error 51 us; skew 500 ppm
I20260929 20:27:21.698312 2447 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:27:21 dist-test-slave-zbnc krb5kdc[2341](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713641, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.53.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:27:21.702016 2447 init.cc:377] Logged in from keytab as kudu/127.17.53.65@KRBTEST.COM (short username kudu)
I20260929 20:27:21.703868 2447 webserver.cc:460] Webserver started at http://127.17.53.65:41193/ using document root <none> and password file <none>
I20260929 20:27:21.704632 2447 fs_manager.cc:359] Metadata directory not provided
I20260929 20:27:21.704789 2447 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:27:21.705157 2447 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:27:21.708472 2447 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790713339161492-17621-0/minicluster-data/ts-0/data/instance:
uuid: "2ae379b0bb064c688d7b22d223e956c5"
format_stamp: "Formatted at 2026-09-29 20:27:21 on dist-test-slave-zbnc"
server_key: "9d428e372b0648d5b6d05e8c0d94df48"
server_key_iv: "427869f0ed5c4d6091eca87da5537f15"
server_key_version: "encryptionkey@0"
I20260929 20:27:21.709304 2447 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790713339161492-17621-0/minicluster-data/ts-0/wal/instance:
uuid: "2ae379b0bb064c688d7b22d223e956c5"
format_stamp: "Formatted at 2026-09-29 20:27:21 on dist-test-slave-zbnc"
server_key: "9d428e372b0648d5b6d05e8c0d94df48"
server_key_iv: "427869f0ed5c4d6091eca87da5537f15"
server_key_version: "encryptionkey@0"
I20260929 20:27:21.715072 2447 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.007s
I20260929 20:27:21.719291 2462 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:27:21.721082 2447 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260929 20:27:21.721329 2447 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790713339161492-17621-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790713339161492-17621-0/minicluster-data/ts-0/wal
uuid: "2ae379b0bb064c688d7b22d223e956c5"
format_stamp: "Formatted at 2026-09-29 20:27:21 on dist-test-slave-zbnc"
server_key: "9d428e372b0648d5b6d05e8c0d94df48"
server_key_iv: "427869f0ed5c4d6091eca87da5537f15"
server_key_version: "encryptionkey@0"
I20260929 20:27:21.721577 2447 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790713339161492-17621-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790713339161492-17621-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790713339161492-17621-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:27:21.738716 2447 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:27:21.743782 2447 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:27:21.744133 2447 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:27:21.745332 2447 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:27:21.746829 2447 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:27:21.747023 2447 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20260929 20:27:21.747135 2447 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:27:21.747190 2447 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:27:21.783948 2447 rpc_server.cc:307] RPC server started. Bound to: 127.17.53.65:44837
I20260929 20:27:21.783994 2574 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.53.65:44837 every 8 connection(s)
I20260929 20:27:21.785352 2447 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790713339161492-17621-0/minicluster-data/ts-0/data/info.pb
I20260929 20:27:21.791424 17621 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu as pid 2447
I20260929 20:27:21.791656 17621 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790713339161492-17621-0/minicluster-data/ts-0/wal/instance
I20260929 20:27:21.792361 17621 external_mini_cluster.cc:1428] Setting key b768a41d012c62ff9cfa74a627bef562
Sep 29 20:27:21 dist-test-slave-zbnc krb5kdc[2341](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713641, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.53.65@KRBTEST.COM for kudu/127.17.53.126@KRBTEST.COM
I20260929 20:27:21.807694 2577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:27:21.788387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.53.65:50209 (local address 127.17.53.126:38017)
0929 20:27:21.788889 (+ 502us) server_negotiation.cc:206] Beginning negotiation
0929 20:27:21.788894 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:27:21.789750 (+ 856us) server_negotiation.cc:407] Connection header received
0929 20:27:21.791948 (+ 2198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:27:21.791955 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:27:21.792056 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:27:21.792193 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:27:21.794145 (+ 1952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:27:21.795061 (+ 916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:27:21.796711 (+ 1650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:27:21.797127 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:27:21.797283 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:27:21.801221 (+ 3938us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:27:21.801227 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:27:21.801230 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:27:21.801264 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:27:21.804079 (+ 2815us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:27:21.805095 (+ 1016us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:27:21.805099 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:27:21.805102 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:27:21.805223 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:27:21.805789 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:27:21.805793 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:27:21.805796 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:27:21.806089 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:27:21.806222 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0929 20:27:21.807235 (+ 1013us) server_negotiation.cc:299] Negotiation successful
0929 20:27:21.807431 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":380,"thread_start_us":267,"threads_started":1}
I20260929 20:27:21.809417 2578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:27:21.788828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:38017 (local address 127.17.53.65:50209)
0929 20:27:21.789573 (+ 745us) negotiation.cc:107] Waiting for socket to connect
0929 20:27:21.789627 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0929 20:27:21.791625 (+ 1998us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:27:21.792484 (+ 859us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:27:21.792496 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:27:21.793194 (+ 698us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:27:21.793965 (+ 771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:27:21.793986 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:27:21.795234 (+ 1248us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:27:21.795247 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:27:21.796551 (+ 1304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:27:21.796564 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:27:21.797386 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:27:21.797395 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:27:21.797666 (+ 271us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:27:21.798422 (+ 756us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:27:21.798447 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:27:21.800978 (+ 2531us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:27:21.804233 (+ 3255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:27:21.804245 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:27:21.804266 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:27:21.804953 (+ 687us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:27:21.805351 (+ 398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:27:21.805359 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:27:21.805366 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:27:21.805639 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:27:21.806244 (+ 605us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:27:21.806255 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:27:21.806751 (+ 496us) client_negotiation.cc:769] Sending connection context
0929 20:27:21.807115 (+ 364us) client_negotiation.cc:240] Negotiation successful
0929 20:27:21.807737 (+ 622us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":497,"thread_start_us":280,"threads_started":1}
I20260929 20:27:21.811093 2575 heartbeater.cc:347] Connected to a master server at 127.17.53.126:38017
I20260929 20:27:21.811400 2575 heartbeater.cc:464] Registering TS with master...
I20260929 20:27:21.812144 2575 heartbeater.cc:511] Master 127.17.53.126:38017 requested a full tablet report, sending...
I20260929 20:27:21.814339 2386 ts_manager.cc:194] Registered new tserver with Master: 2ae379b0bb064c688d7b22d223e956c5 (127.17.53.65:44837)
I20260929 20:27:21.815860 2386 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.53.65@KRBTEST.COM'} at 127.17.53.65:50209
WARNING: no policy specified for kudu/127.17.53.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.53.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.53.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.66.keytab.
Entry for principal kudu/127.17.53.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.66.keytab.
I20260929 20:27:21.850035 17621 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
/tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790713339161492-17621-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790713339161492-17621-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790713339161492-17621-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790713339161492-17621-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.53.66:0
--local_ip_for_outbound_sockets=127.17.53.66
--webserver_interface=127.17.53.66
--webserver_port=0
--tserver_master_addrs=127.17.53.126:38017
--keytab_file=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.66.keytab
--principal=kudu/127.17.53.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.53.84:33833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790713339161492-17621-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790713339161492-17621-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790713339161492-17621-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:27:22.183718 2581 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:27:22.183993 2581 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:27:22.184177 2581 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:27:22.191648 2581 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:27:22.191756 2581 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:27:22.191962 2581 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.53.66
I20260929 20:27:22.201795 2581 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.53.84:33833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790713339161492-17621-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790713339161492-17621-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.66.keytab
--principal=kudu/127.17.53.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.53.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790713339161492-17621-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.17.53.66
--webserver_port=0
--tserver_master_addrs=127.17.53.126:38017
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.53.66
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790713339161492-17621-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:27:22.203274 2581 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:27:22.204933 2581 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:27:22.217917 2586 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:27:22.217917 2589 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:27:22.218035 2587 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:27:22.219520 2581 server_base.cc:1034] running on GCE node
I20260929 20:27:22.220192 2581 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:27:22.221220 2581 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:27:22.222496 2581 hybrid_clock.cc:648] HybridClock initialized: now 1790713642222448 us; error 32 us; skew 500 ppm
I20260929 20:27:22.223031 2581 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:27:22 dist-test-slave-zbnc krb5kdc[2341](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713642, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.53.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:27:22.226632 2581 init.cc:377] Logged in from keytab as kudu/127.17.53.66@KRBTEST.COM (short username kudu)
I20260929 20:27:22.228318 2581 webserver.cc:460] Webserver started at http://127.17.53.66:43995/ using document root <none> and password file <none>
I20260929 20:27:22.228994 2581 fs_manager.cc:359] Metadata directory not provided
I20260929 20:27:22.229107 2581 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:27:22.229511 2581 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:27:22.232785 2581 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790713339161492-17621-0/minicluster-data/ts-1/data/instance:
uuid: "d72848c751d44ffda4bb13aff2bc4b8b"
format_stamp: "Formatted at 2026-09-29 20:27:22 on dist-test-slave-zbnc"
server_key: "8e2cb7aca0517396b9db8ecd22c80df9"
server_key_iv: "db0ab127ace37d4891ac53b88c981dc6"
server_key_version: "encryptionkey@0"
I20260929 20:27:22.233647 2581 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790713339161492-17621-0/minicluster-data/ts-1/wal/instance:
uuid: "d72848c751d44ffda4bb13aff2bc4b8b"
format_stamp: "Formatted at 2026-09-29 20:27:22 on dist-test-slave-zbnc"
server_key: "8e2cb7aca0517396b9db8ecd22c80df9"
server_key_iv: "db0ab127ace37d4891ac53b88c981dc6"
server_key_version: "encryptionkey@0"
I20260929 20:27:22.239447 2581 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260929 20:27:22.242929 2596 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:27:22.244298 2581 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 20:27:22.244490 2581 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790713339161492-17621-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790713339161492-17621-0/minicluster-data/ts-1/wal
uuid: "d72848c751d44ffda4bb13aff2bc4b8b"
format_stamp: "Formatted at 2026-09-29 20:27:22 on dist-test-slave-zbnc"
server_key: "8e2cb7aca0517396b9db8ecd22c80df9"
server_key_iv: "db0ab127ace37d4891ac53b88c981dc6"
server_key_version: "encryptionkey@0"
I20260929 20:27:22.244722 2581 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790713339161492-17621-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790713339161492-17621-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790713339161492-17621-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:27:22.296939 2581 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:27:22.301523 2581 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:27:22.301862 2581 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:27:22.302922 2581 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:27:22.304464 2581 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:27:22.304589 2581 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:27:22.304764 2581 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:27:22.304982 2581 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:27:22.339313 2581 rpc_server.cc:307] RPC server started. Bound to: 127.17.53.66:34981
I20260929 20:27:22.339366 2708 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.53.66:34981 every 8 connection(s)
I20260929 20:27:22.340703 2581 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790713339161492-17621-0/minicluster-data/ts-1/data/info.pb
I20260929 20:27:22.345458 17621 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu as pid 2581
I20260929 20:27:22.345646 17621 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790713339161492-17621-0/minicluster-data/ts-1/wal/instance
I20260929 20:27:22.346221 17621 external_mini_cluster.cc:1428] Setting key a4069d868a7b59bc93f1a4e708e227d3
Sep 29 20:27:22 dist-test-slave-zbnc krb5kdc[2341](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713642, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.53.66@KRBTEST.COM for kudu/127.17.53.126@KRBTEST.COM
I20260929 20:27:22.362689 2711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:27:22.343357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.53.66:51805 (local address 127.17.53.126:38017)
0929 20:27:22.343844 (+ 487us) server_negotiation.cc:206] Beginning negotiation
0929 20:27:22.343849 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:27:22.344642 (+ 793us) server_negotiation.cc:407] Connection header received
0929 20:27:22.347246 (+ 2604us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:27:22.347251 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:27:22.347341 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:27:22.347472 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:27:22.349381 (+ 1909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:27:22.350202 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:27:22.351950 (+ 1748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:27:22.352429 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:27:22.352591 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:27:22.356351 (+ 3760us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:27:22.356358 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:27:22.356364 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:27:22.356409 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:27:22.359259 (+ 2850us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:27:22.360259 (+ 1000us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:27:22.360262 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:27:22.360265 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:27:22.360395 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:27:22.360940 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:27:22.360943 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:27:22.360945 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:27:22.361197 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:27:22.361310 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0929 20:27:22.362242 (+ 932us) server_negotiation.cc:299] Negotiation successful
0929 20:27:22.362417 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":390,"thread_start_us":263,"threads_started":1}
I20260929 20:27:22.364427 2712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:27:22.343822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:38017 (local address 127.17.53.66:51805)
0929 20:27:22.344480 (+ 658us) negotiation.cc:107] Waiting for socket to connect
0929 20:27:22.344524 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0929 20:27:22.346984 (+ 2460us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:27:22.347677 (+ 693us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:27:22.347686 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:27:22.348430 (+ 744us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:27:22.349188 (+ 758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:27:22.349210 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:27:22.350373 (+ 1163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:27:22.350381 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:27:22.351760 (+ 1379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:27:22.351773 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:27:22.352690 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:27:22.352699 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:27:22.352966 (+ 267us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:27:22.353696 (+ 730us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:27:22.353726 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:27:22.356093 (+ 2367us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:27:22.359402 (+ 3309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:27:22.359414 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:27:22.359434 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:27:22.360116 (+ 682us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:27:22.360514 (+ 398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:27:22.360522 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:27:22.360529 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:27:22.360809 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:27:22.361339 (+ 530us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:27:22.361350 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:27:22.361826 (+ 476us) client_negotiation.cc:769] Sending connection context
0929 20:27:22.362168 (+ 342us) client_negotiation.cc:240] Negotiation successful
0929 20:27:22.362825 (+ 657us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":435,"thread_start_us":265,"threads_started":1}
I20260929 20:27:22.366050 2709 heartbeater.cc:347] Connected to a master server at 127.17.53.126:38017
I20260929 20:27:22.366357 2709 heartbeater.cc:464] Registering TS with master...
I20260929 20:27:22.367136 2709 heartbeater.cc:511] Master 127.17.53.126:38017 requested a full tablet report, sending...
I20260929 20:27:22.368831 2386 ts_manager.cc:194] Registered new tserver with Master: d72848c751d44ffda4bb13aff2bc4b8b (127.17.53.66:34981)
I20260929 20:27:22.369688 2386 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.53.66@KRBTEST.COM'} at 127.17.53.66:51805
WARNING: no policy specified for kudu/127.17.53.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.53.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.53.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.67.keytab.
Entry for principal kudu/127.17.53.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.67.keytab.
I20260929 20:27:22.407357 17621 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
/tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790713339161492-17621-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790713339161492-17621-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790713339161492-17621-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790713339161492-17621-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.53.67:0
--local_ip_for_outbound_sockets=127.17.53.67
--webserver_interface=127.17.53.67
--webserver_port=0
--tserver_master_addrs=127.17.53.126:38017
--keytab_file=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.67.keytab
--principal=kudu/127.17.53.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.53.84:33833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790713339161492-17621-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790713339161492-17621-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790713339161492-17621-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:27:22.764494 2715 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:27:22.764832 2715 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:27:22.765013 2715 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:27:22.772521 2715 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:27:22.772698 2715 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:27:22.772884 2715 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.53.67
I20260929 20:27:22.782925 2715 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.53.84:33833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790713339161492-17621-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790713339161492-17621-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.67.keytab
--principal=kudu/127.17.53.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.53.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790713339161492-17621-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.17.53.67
--webserver_port=0
--tserver_master_addrs=127.17.53.126:38017
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.53.67
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790713339161492-17621-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:27:22.784335 2715 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:27:22.785638 2715 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:27:22.799559 2720 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:27:22.799510 2723 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:27:22.799576 2721 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:27:22.800503 2715 server_base.cc:1034] running on GCE node
I20260929 20:27:22.801110 2715 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:27:22.802011 2715 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:27:22.803239 2715 hybrid_clock.cc:648] HybridClock initialized: now 1790713642803147 us; error 58 us; skew 500 ppm
I20260929 20:27:22.803555 2715 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:27:22 dist-test-slave-zbnc krb5kdc[2341](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713642, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.53.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:27:22.806947 2715 init.cc:377] Logged in from keytab as kudu/127.17.53.67@KRBTEST.COM (short username kudu)
I20260929 20:27:22.808382 2715 webserver.cc:460] Webserver started at http://127.17.53.67:45179/ using document root <none> and password file <none>
I20260929 20:27:22.809044 2715 fs_manager.cc:359] Metadata directory not provided
I20260929 20:27:22.809147 2715 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:27:22.809408 2715 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:27:22.812431 2715 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790713339161492-17621-0/minicluster-data/ts-2/data/instance:
uuid: "f2efee3001a740efaf8c26ef3cc7958a"
format_stamp: "Formatted at 2026-09-29 20:27:22 on dist-test-slave-zbnc"
server_key: "5a8ff5ba23f770532f4ce3d5ce0f049a"
server_key_iv: "3a044105bf38cd897b5bf5f2551925e1"
server_key_version: "encryptionkey@0"
I20260929 20:27:22.813311 2715 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790713339161492-17621-0/minicluster-data/ts-2/wal/instance:
uuid: "f2efee3001a740efaf8c26ef3cc7958a"
format_stamp: "Formatted at 2026-09-29 20:27:22 on dist-test-slave-zbnc"
server_key: "5a8ff5ba23f770532f4ce3d5ce0f049a"
server_key_iv: "3a044105bf38cd897b5bf5f2551925e1"
server_key_version: "encryptionkey@0"
I20260929 20:27:22.818411 2715 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 20:27:22.818974 2575 heartbeater.cc:503] Master 127.17.53.126:38017 was elected leader, sending a full tablet report...
I20260929 20:27:22.822510 2730 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:27:22.823972 2715 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 20:27:22.824106 2715 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790713339161492-17621-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790713339161492-17621-0/minicluster-data/ts-2/wal
uuid: "f2efee3001a740efaf8c26ef3cc7958a"
format_stamp: "Formatted at 2026-09-29 20:27:22 on dist-test-slave-zbnc"
server_key: "5a8ff5ba23f770532f4ce3d5ce0f049a"
server_key_iv: "3a044105bf38cd897b5bf5f2551925e1"
server_key_version: "encryptionkey@0"
I20260929 20:27:22.824282 2715 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790713339161492-17621-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790713339161492-17621-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790713339161492-17621-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:27:22.846715 2715 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:27:22.852088 2715 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:27:22.852406 2715 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:27:22.853557 2715 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:27:22.855094 2715 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:27:22.855187 2715 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:27:22.855301 2715 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:27:22.855360 2715 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:27:22.896653 2715 rpc_server.cc:307] RPC server started. Bound to: 127.17.53.67:39253
I20260929 20:27:22.896677 2842 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.53.67:39253 every 8 connection(s)
I20260929 20:27:22.898181 2715 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790713339161492-17621-0/minicluster-data/ts-2/data/info.pb
I20260929 20:27:22.902176 17621 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu as pid 2715
I20260929 20:27:22.902436 17621 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790713339161492-17621-0/minicluster-data/ts-2/wal/instance
I20260929 20:27:22.903087 17621 external_mini_cluster.cc:1428] Setting key 70a5df9009dd5a790566c9ffe4252eb0
Sep 29 20:27:22 dist-test-slave-zbnc krb5kdc[2341](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713642, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.53.67@KRBTEST.COM for kudu/127.17.53.126@KRBTEST.COM
I20260929 20:27:22.920584 2845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:27:22.901072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.53.67:41491 (local address 127.17.53.126:38017)
0929 20:27:22.901639 (+ 567us) server_negotiation.cc:206] Beginning negotiation
0929 20:27:22.901647 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 20:27:22.902930 (+ 1283us) server_negotiation.cc:407] Connection header received
0929 20:27:22.905619 (+ 2689us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:27:22.905624 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:27:22.905849 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:27:22.905992 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:27:22.907846 (+ 1854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:27:22.908654 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:27:22.910170 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:27:22.910582 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:27:22.910677 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:27:22.914271 (+ 3594us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:27:22.914276 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:27:22.914279 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:27:22.914314 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:27:22.916905 (+ 2591us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:27:22.917908 (+ 1003us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:27:22.917912 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:27:22.917914 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:27:22.918112 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:27:22.918670 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:27:22.918675 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:27:22.918678 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:27:22.919162 (+ 484us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:27:22.919327 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0929 20:27:22.920144 (+ 817us) server_negotiation.cc:299] Negotiation successful
0929 20:27:22.920311 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":413,"thread_start_us":253,"threads_started":1}
I20260929 20:27:22.921535 2846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:27:22.901660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:38017 (local address 127.17.53.67:41491)
0929 20:27:22.902692 (+ 1032us) negotiation.cc:107] Waiting for socket to connect
0929 20:27:22.902745 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0929 20:27:22.905269 (+ 2524us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:27:22.906306 (+ 1037us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:27:22.906314 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:27:22.907059 (+ 745us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:27:22.907665 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:27:22.907683 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:27:22.908783 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:27:22.908787 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:27:22.910036 (+ 1249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:27:22.910048 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:27:22.910760 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:27:22.910764 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:27:22.911050 (+ 286us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:27:22.911716 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:27:22.911738 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:27:22.914047 (+ 2309us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:27:22.917051 (+ 3004us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:27:22.917058 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:27:22.917069 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:27:22.917685 (+ 616us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:27:22.918258 (+ 573us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:27:22.918263 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:27:22.918266 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:27:22.918519 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:27:22.919314 (+ 795us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:27:22.919325 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:27:22.919770 (+ 445us) client_negotiation.cc:769] Sending connection context
0929 20:27:22.920018 (+ 248us) client_negotiation.cc:240] Negotiation successful
0929 20:27:22.920404 (+ 386us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":700,"thread_start_us":351,"threads_started":1}
I20260929 20:27:22.923053 2843 heartbeater.cc:347] Connected to a master server at 127.17.53.126:38017
I20260929 20:27:22.923353 2843 heartbeater.cc:464] Registering TS with master...
I20260929 20:27:22.924088 2843 heartbeater.cc:511] Master 127.17.53.126:38017 requested a full tablet report, sending...
I20260929 20:27:22.925655 2386 ts_manager.cc:194] Registered new tserver with Master: f2efee3001a740efaf8c26ef3cc7958a (127.17.53.67:39253)
I20260929 20:27:22.926553 2386 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.53.67@KRBTEST.COM'} at 127.17.53.67:41491
I20260929 20:27:22.932185 17621 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 20:27:22.950527 2845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:27:22.937131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50004 (local address 127.17.53.126:38017)
0929 20:27:22.937365 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0929 20:27:22.937375 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 20:27:22.937694 (+ 319us) server_negotiation.cc:407] Connection header received
0929 20:27:22.937995 (+ 301us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:27:22.937999 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:27:22.938073 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:27:22.938284 (+ 211us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:27:22.939842 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:27:22.940586 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:27:22.941977 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:27:22.942397 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:27:22.942540 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:27:22.945070 (+ 2530us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:27:22.945074 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:27:22.945077 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:27:22.945110 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:27:22.947697 (+ 2587us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:27:22.948436 (+ 739us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:27:22.948443 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:27:22.948451 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:27:22.948571 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:27:22.948952 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:27:22.948956 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:27:22.948960 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:27:22.949357 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:27:22.949613 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0929 20:27:22.949965 (+ 352us) server_negotiation.cc:299] Negotiation successful
0929 20:27:22.950188 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260929 20:27:22.954800 2386 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50004:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 20:27:22.962113 2386 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 20:27:23.001159 2857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:27:22.974260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.66:34981 (local address 127.0.0.1:59256)
0929 20:27:22.986962 (+ 12702us) negotiation.cc:107] Waiting for socket to connect
0929 20:27:22.987020 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0929 20:27:22.987266 (+ 246us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:27:22.993609 (+ 6343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:27:22.993622 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:27:22.993666 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:27:22.994151 (+ 485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:27:22.994166 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:27:22.996337 (+ 2171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:27:22.996342 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:27:22.998719 (+ 2377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:27:22.998730 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:27:23.000134 (+ 1404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:27:23.000140 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:27:23.000336 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:27:23.000391 (+ 55us) client_negotiation.cc:769] Sending connection context
0929 20:27:23.000567 (+ 176us) client_negotiation.cc:240] Negotiation successful
0929 20:27:23.000765 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12320,"thread_start_us":14803,"threads_started":1}
I20260929 20:27:23.002100 2858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:27:22.974382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59256 (local address 127.17.53.66:34981)
0929 20:27:22.991066 (+ 16684us) server_negotiation.cc:206] Beginning negotiation
0929 20:27:22.991076 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 20:27:22.991129 (+ 53us) server_negotiation.cc:407] Connection header received
0929 20:27:22.991268 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:27:22.991276 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:27:22.991579 (+ 303us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:27:22.991824 (+ 245us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:27:22.994411 (+ 2587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:27:22.996159 (+ 1748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:27:22.998915 (+ 2756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:27:22.999918 (+ 1003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:27:23.000070 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:27:23.000298 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0929 20:27:23.001602 (+ 1304us) server_negotiation.cc:299] Negotiation successful
0929 20:27:23.001786 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16352,"thread_start_us":16746,"threads_started":1}
I20260929 20:27:23.002995 2861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:27:22.991496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.67:39253 (local address 127.0.0.1:45750)
0929 20:27:22.994621 (+ 3125us) negotiation.cc:107] Waiting for socket to connect
0929 20:27:22.994641 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 20:27:22.994759 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:27:22.995504 (+ 745us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:27:22.995508 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:27:22.995533 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:27:22.996043 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:27:22.996060 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:27:22.997825 (+ 1765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:27:22.997832 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:27:23.000030 (+ 2198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:27:23.000045 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:27:23.002089 (+ 2044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:27:23.002097 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:27:23.002286 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:27:23.002332 (+ 46us) client_negotiation.cc:769] Sending connection context
0929 20:27:23.002433 (+ 101us) client_negotiation.cc:240] Negotiation successful
0929 20:27:23.002595 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2974,"mutex_wait_us":4,"thread_start_us":2821,"threads_started":1}
I20260929 20:27:23.003407 2859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:27:22.974729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34046 (local address 127.17.53.65:44837)
0929 20:27:22.994998 (+ 20269us) server_negotiation.cc:206] Beginning negotiation
0929 20:27:22.995007 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 20:27:22.995036 (+ 29us) server_negotiation.cc:407] Connection header received
0929 20:27:22.995170 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:27:22.995178 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:27:22.995411 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:27:22.996376 (+ 965us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:27:22.996674 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:27:22.998244 (+ 1570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:27:23.000799 (+ 2555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:27:23.002274 (+ 1475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:27:23.002440 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:27:23.002675 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0929 20:27:23.002923 (+ 248us) server_negotiation.cc:299] Negotiation successful
0929 20:27:23.003080 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20054,"thread_start_us":19866,"threads_started":1}
I20260929 20:27:23.003823 2860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:27:22.987984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45750 (local address 127.17.53.67:39253)
0929 20:27:22.993387 (+ 5403us) server_negotiation.cc:206] Beginning negotiation
0929 20:27:22.993397 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 20:27:22.994909 (+ 1512us) server_negotiation.cc:407] Connection header received
0929 20:27:22.995050 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:27:22.995057 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:27:22.995315 (+ 258us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:27:22.995529 (+ 214us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:27:22.996315 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:27:22.997684 (+ 1369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:27:23.000217 (+ 2533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:27:23.001881 (+ 1664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:27:23.002022 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:27:23.002243 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0929 20:27:23.003277 (+ 1034us) server_negotiation.cc:299] Negotiation successful
0929 20:27:23.003498 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5199,"thread_start_us":5887,"threads_started":1}
I20260929 20:27:23.004042 2856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:27:22.974355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.65:44837 (local address 127.0.0.1:34046)
0929 20:27:22.988808 (+ 14453us) negotiation.cc:107] Waiting for socket to connect
0929 20:27:22.988831 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0929 20:27:22.988972 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:27:22.995692 (+ 6720us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:27:22.995698 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:27:22.995724 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:27:22.996177 (+ 453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:27:22.996190 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:27:22.998402 (+ 2212us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:27:22.998407 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:27:23.000607 (+ 2200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:27:23.000622 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:27:23.002506 (+ 1884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:27:23.002512 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:27:23.002702 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:27:23.002733 (+ 31us) client_negotiation.cc:769] Sending connection context
0929 20:27:23.002854 (+ 121us) client_negotiation.cc:240] Negotiation successful
0929 20:27:23.003006 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14295,"mutex_wait_us":3,"thread_start_us":12037,"threads_started":1}
I20260929 20:27:23.006839 2644 tablet_service.cc:1467] Processing CreateTablet for tablet 422f3ab34bc047a1b3d38845a758377b (DEFAULT_TABLE table=test-table [id=0c476a645cb4432b85c547e8b49aa58d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:27:23.008154 2644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 422f3ab34bc047a1b3d38845a758377b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:27:23.007977 2778 tablet_service.cc:1467] Processing CreateTablet for tablet 422f3ab34bc047a1b3d38845a758377b (DEFAULT_TABLE table=test-table [id=0c476a645cb4432b85c547e8b49aa58d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:27:23.008556 2510 tablet_service.cc:1467] Processing CreateTablet for tablet 422f3ab34bc047a1b3d38845a758377b (DEFAULT_TABLE table=test-table [id=0c476a645cb4432b85c547e8b49aa58d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:27:23.009450 2778 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 422f3ab34bc047a1b3d38845a758377b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:27:23.009924 2510 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 422f3ab34bc047a1b3d38845a758377b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:27:23.020241 2862 tablet_bootstrap.cc:492] T 422f3ab34bc047a1b3d38845a758377b P d72848c751d44ffda4bb13aff2bc4b8b: Bootstrap starting.
I20260929 20:27:23.023455 2862 tablet_bootstrap.cc:654] T 422f3ab34bc047a1b3d38845a758377b P d72848c751d44ffda4bb13aff2bc4b8b: Neither blocks nor log segments found. Creating new log.
I20260929 20:27:23.024295 2863 tablet_bootstrap.cc:492] T 422f3ab34bc047a1b3d38845a758377b P 2ae379b0bb064c688d7b22d223e956c5: Bootstrap starting.
I20260929 20:27:23.024806 2862 log.cc:824] T 422f3ab34bc047a1b3d38845a758377b P d72848c751d44ffda4bb13aff2bc4b8b: Log is configured to *not* fsync() on all Append() calls
I20260929 20:27:23.025171 2864 tablet_bootstrap.cc:492] T 422f3ab34bc047a1b3d38845a758377b P f2efee3001a740efaf8c26ef3cc7958a: Bootstrap starting.
I20260929 20:27:23.028898 2864 tablet_bootstrap.cc:654] T 422f3ab34bc047a1b3d38845a758377b P f2efee3001a740efaf8c26ef3cc7958a: Neither blocks nor log segments found. Creating new log.
I20260929 20:27:23.028929 2863 tablet_bootstrap.cc:654] T 422f3ab34bc047a1b3d38845a758377b P 2ae379b0bb064c688d7b22d223e956c5: Neither blocks nor log segments found. Creating new log.
I20260929 20:27:23.029508 2862 tablet_bootstrap.cc:492] T 422f3ab34bc047a1b3d38845a758377b P d72848c751d44ffda4bb13aff2bc4b8b: No bootstrap required, opened a new log
I20260929 20:27:23.029783 2862 ts_tablet_manager.cc:1397] T 422f3ab34bc047a1b3d38845a758377b P d72848c751d44ffda4bb13aff2bc4b8b: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.001s
I20260929 20:27:23.030072 2863 log.cc:824] T 422f3ab34bc047a1b3d38845a758377b P 2ae379b0bb064c688d7b22d223e956c5: Log is configured to *not* fsync() on all Append() calls
I20260929 20:27:23.030161 2864 log.cc:824] T 422f3ab34bc047a1b3d38845a758377b P f2efee3001a740efaf8c26ef3cc7958a: Log is configured to *not* fsync() on all Append() calls
I20260929 20:27:23.032754 2863 tablet_bootstrap.cc:492] T 422f3ab34bc047a1b3d38845a758377b P 2ae379b0bb064c688d7b22d223e956c5: No bootstrap required, opened a new log
I20260929 20:27:23.033025 2863 ts_tablet_manager.cc:1397] T 422f3ab34bc047a1b3d38845a758377b P 2ae379b0bb064c688d7b22d223e956c5: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.006s
I20260929 20:27:23.036870 2864 tablet_bootstrap.cc:492] T 422f3ab34bc047a1b3d38845a758377b P f2efee3001a740efaf8c26ef3cc7958a: No bootstrap required, opened a new log
I20260929 20:27:23.037135 2864 ts_tablet_manager.cc:1397] T 422f3ab34bc047a1b3d38845a758377b P f2efee3001a740efaf8c26ef3cc7958a: Time spent bootstrapping tablet: real 0.012s user 0.000s sys 0.006s
I20260929 20:27:23.037803 2862 raft_consensus.cc:348] T 422f3ab34bc047a1b3d38845a758377b P d72848c751d44ffda4bb13aff2bc4b8b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d72848c751d44ffda4bb13aff2bc4b8b" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 34981 } } peers { permanent_uuid: "2ae379b0bb064c688d7b22d223e956c5" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 44837 } } peers { permanent_uuid: "f2efee3001a740efaf8c26ef3cc7958a" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 39253 } }
I20260929 20:27:23.038370 2862 raft_consensus.cc:374] T 422f3ab34bc047a1b3d38845a758377b P d72848c751d44ffda4bb13aff2bc4b8b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:27:23.038524 2862 raft_consensus.cc:729] T 422f3ab34bc047a1b3d38845a758377b P d72848c751d44ffda4bb13aff2bc4b8b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d72848c751d44ffda4bb13aff2bc4b8b, State: Initialized, Role: FOLLOWER
I20260929 20:27:23.039299 2862 consensus_queue.cc:260] T 422f3ab34bc047a1b3d38845a758377b P d72848c751d44ffda4bb13aff2bc4b8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d72848c751d44ffda4bb13aff2bc4b8b" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 34981 } } peers { permanent_uuid: "2ae379b0bb064c688d7b22d223e956c5" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 44837 } } peers { permanent_uuid: "f2efee3001a740efaf8c26ef3cc7958a" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 39253 } }
I20260929 20:27:23.040432 2709 heartbeater.cc:503] Master 127.17.53.126:38017 was elected leader, sending a full tablet report...
I20260929 20:27:23.041045 2862 ts_tablet_manager.cc:1428] T 422f3ab34bc047a1b3d38845a758377b P d72848c751d44ffda4bb13aff2bc4b8b: Time spent starting tablet: real 0.011s user 0.003s sys 0.008s
I20260929 20:27:23.043213 2863 raft_consensus.cc:348] T 422f3ab34bc047a1b3d38845a758377b P 2ae379b0bb064c688d7b22d223e956c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d72848c751d44ffda4bb13aff2bc4b8b" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 34981 } } peers { permanent_uuid: "2ae379b0bb064c688d7b22d223e956c5" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 44837 } } peers { permanent_uuid: "f2efee3001a740efaf8c26ef3cc7958a" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 39253 } }
I20260929 20:27:23.043642 2863 raft_consensus.cc:374] T 422f3ab34bc047a1b3d38845a758377b P 2ae379b0bb064c688d7b22d223e956c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:27:23.043742 2863 raft_consensus.cc:729] T 422f3ab34bc047a1b3d38845a758377b P 2ae379b0bb064c688d7b22d223e956c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ae379b0bb064c688d7b22d223e956c5, State: Initialized, Role: FOLLOWER
I20260929 20:27:23.044375 2863 consensus_queue.cc:260] T 422f3ab34bc047a1b3d38845a758377b P 2ae379b0bb064c688d7b22d223e956c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d72848c751d44ffda4bb13aff2bc4b8b" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 34981 } } peers { permanent_uuid: "2ae379b0bb064c688d7b22d223e956c5" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 44837 } } peers { permanent_uuid: "f2efee3001a740efaf8c26ef3cc7958a" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 39253 } }
I20260929 20:27:23.046109 2863 ts_tablet_manager.cc:1428] T 422f3ab34bc047a1b3d38845a758377b P 2ae379b0bb064c688d7b22d223e956c5: Time spent starting tablet: real 0.013s user 0.014s sys 0.000s
I20260929 20:27:23.047430 2864 raft_consensus.cc:348] T 422f3ab34bc047a1b3d38845a758377b P f2efee3001a740efaf8c26ef3cc7958a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d72848c751d44ffda4bb13aff2bc4b8b" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 34981 } } peers { permanent_uuid: "2ae379b0bb064c688d7b22d223e956c5" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 44837 } } peers { permanent_uuid: "f2efee3001a740efaf8c26ef3cc7958a" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 39253 } }
I20260929 20:27:23.047777 2864 raft_consensus.cc:374] T 422f3ab34bc047a1b3d38845a758377b P f2efee3001a740efaf8c26ef3cc7958a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:27:23.047917 2864 raft_consensus.cc:729] T 422f3ab34bc047a1b3d38845a758377b P f2efee3001a740efaf8c26ef3cc7958a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2efee3001a740efaf8c26ef3cc7958a, State: Initialized, Role: FOLLOWER
I20260929 20:27:23.048504 2864 consensus_queue.cc:260] T 422f3ab34bc047a1b3d38845a758377b P f2efee3001a740efaf8c26ef3cc7958a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d72848c751d44ffda4bb13aff2bc4b8b" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 34981 } } peers { permanent_uuid: "2ae379b0bb064c688d7b22d223e956c5" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 44837 } } peers { permanent_uuid: "f2efee3001a740efaf8c26ef3cc7958a" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 39253 } }
I20260929 20:27:23.049605 2843 heartbeater.cc:503] Master 127.17.53.126:38017 was elected leader, sending a full tablet report...
I20260929 20:27:23.050216 2864 ts_tablet_manager.cc:1428] T 422f3ab34bc047a1b3d38845a758377b P f2efee3001a740efaf8c26ef3cc7958a: Time spent starting tablet: real 0.013s user 0.008s sys 0.006s
W20260929 20:27:23.092748 2710 tablet.cc:2369] T 422f3ab34bc047a1b3d38845a758377b P d72848c751d44ffda4bb13aff2bc4b8b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 20:27:23.150120 2844 tablet.cc:2369] T 422f3ab34bc047a1b3d38845a758377b P f2efee3001a740efaf8c26ef3cc7958a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 20:27:23.288127 2576 tablet.cc:2369] T 422f3ab34bc047a1b3d38845a758377b P 2ae379b0bb064c688d7b22d223e956c5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 20:27:23.290442 2870 raft_consensus.cc:482] T 422f3ab34bc047a1b3d38845a758377b P f2efee3001a740efaf8c26ef3cc7958a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 20:27:23.290752 2870 raft_consensus.cc:504] T 422f3ab34bc047a1b3d38845a758377b P f2efee3001a740efaf8c26ef3cc7958a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d72848c751d44ffda4bb13aff2bc4b8b" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 34981 } } peers { permanent_uuid: "2ae379b0bb064c688d7b22d223e956c5" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 44837 } } peers { permanent_uuid: "f2efee3001a740efaf8c26ef3cc7958a" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 39253 } }
I20260929 20:27:23.292750 2870 leader_election.cc:288] T 422f3ab34bc047a1b3d38845a758377b P f2efee3001a740efaf8c26ef3cc7958a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d72848c751d44ffda4bb13aff2bc4b8b (127.17.53.66:34981), 2ae379b0bb064c688d7b22d223e956c5 (127.17.53.65:44837)
I20260929 20:27:23.298877 2846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:27:23.292920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.66:34981 (local address 127.17.53.67:49459)
0929 20:27:23.293123 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0929 20:27:23.293147 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0929 20:27:23.293297 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:27:23.294018 (+ 721us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:27:23.294026 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:27:23.294061 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:27:23.294301 (+ 240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:27:23.294319 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:27:23.295578 (+ 1259us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:27:23.295583 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:27:23.296928 (+ 1345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:27:23.296940 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:27:23.298081 (+ 1141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:27:23.298087 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:27:23.298287 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:27:23.298318 (+ 31us) client_negotiation.cc:769] Sending connection context
0929 20:27:23.298424 (+ 106us) client_negotiation.cc:240] Negotiation successful
0929 20:27:23.298556 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43,"mutex_wait_us":32}
I20260929 20:27:23.298875 2858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:27:23.293061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.53.67:49459 (local address 127.17.53.66:34981)
0929 20:27:23.293287 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0929 20:27:23.293294 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:27:23.293327 (+ 33us) server_negotiation.cc:407] Connection header received
0929 20:27:23.293490 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:27:23.293499 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:27:23.293601 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:27:23.293767 (+ 166us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:27:23.294488 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:27:23.295433 (+ 945us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:27:23.297086 (+ 1653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:27:23.297913 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:27:23.298088 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:27:23.298162 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0929 20:27:23.298451 (+ 289us) server_negotiation.cc:299] Negotiation successful
0929 20:27:23.298577 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260929 20:27:23.299713 2859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:27:23.293389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.53.67:42523 (local address 127.17.53.65:44837)
0929 20:27:23.293867 (+ 478us) server_negotiation.cc:206] Beginning negotiation
0929 20:27:23.293874 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:27:23.294507 (+ 633us) server_negotiation.cc:407] Connection header received
0929 20:27:23.294786 (+ 279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:27:23.294790 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:27:23.294899 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:27:23.295024 (+ 125us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:27:23.295509 (+ 485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:27:23.296260 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:27:23.297985 (+ 1725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:27:23.298755 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:27:23.298969 (+ 214us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:27:23.299040 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0929 20:27:23.299342 (+ 302us) server_negotiation.cc:299] Negotiation successful
0929 20:27:23.299474 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260929 20:27:23.299821 2871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:27:23.293234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.65:44837 (local address 127.17.53.67:42523)
0929 20:27:23.294399 (+ 1165us) negotiation.cc:107] Waiting for socket to connect
0929 20:27:23.294420 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 20:27:23.294621 (+ 201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:27:23.295022 (+ 401us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:27:23.295027 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:27:23.295049 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:27:23.295363 (+ 314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:27:23.295371 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:27:23.296416 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:27:23.296421 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:27:23.297822 (+ 1401us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:27:23.297833 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:27:23.298985 (+ 1152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:27:23.298991 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:27:23.299154 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:27:23.299182 (+ 28us) client_negotiation.cc:769] Sending connection context
0929 20:27:23.299315 (+ 133us) client_negotiation.cc:240] Negotiation successful
0929 20:27:23.299450 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":996,"mutex_wait_us":20,"thread_start_us":820,"threads_started":1}
I20260929 20:27:23.300045 2664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "422f3ab34bc047a1b3d38845a758377b" candidate_uuid: "f2efee3001a740efaf8c26ef3cc7958a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d72848c751d44ffda4bb13aff2bc4b8b" is_pre_election: true
I20260929 20:27:23.300618 2664 raft_consensus.cc:2454] T 422f3ab34bc047a1b3d38845a758377b P d72848c751d44ffda4bb13aff2bc4b8b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f2efee3001a740efaf8c26ef3cc7958a in term 0.
I20260929 20:27:23.300732 2530 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "422f3ab34bc047a1b3d38845a758377b" candidate_uuid: "f2efee3001a740efaf8c26ef3cc7958a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ae379b0bb064c688d7b22d223e956c5" is_pre_election: true
I20260929 20:27:23.301295 2530 raft_consensus.cc:2454] T 422f3ab34bc047a1b3d38845a758377b P 2ae379b0bb064c688d7b22d223e956c5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f2efee3001a740efaf8c26ef3cc7958a in term 0.
I20260929 20:27:23.301352 2734 leader_election.cc:302] T 422f3ab34bc047a1b3d38845a758377b P f2efee3001a740efaf8c26ef3cc7958a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d72848c751d44ffda4bb13aff2bc4b8b, f2efee3001a740efaf8c26ef3cc7958a; no voters:
I20260929 20:27:23.301786 2870 raft_consensus.cc:2781] T 422f3ab34bc047a1b3d38845a758377b P f2efee3001a740efaf8c26ef3cc7958a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 20:27:23.302013 2870 raft_consensus.cc:482] T 422f3ab34bc047a1b3d38845a758377b P f2efee3001a740efaf8c26ef3cc7958a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 20:27:23.302137 2870 raft_consensus.cc:3037] T 422f3ab34bc047a1b3d38845a758377b P f2efee3001a740efaf8c26ef3cc7958a [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:27:23.304467 2870 raft_consensus.cc:504] T 422f3ab34bc047a1b3d38845a758377b P f2efee3001a740efaf8c26ef3cc7958a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d72848c751d44ffda4bb13aff2bc4b8b" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 34981 } } peers { permanent_uuid: "2ae379b0bb064c688d7b22d223e956c5" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 44837 } } peers { permanent_uuid: "f2efee3001a740efaf8c26ef3cc7958a" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 39253 } }
I20260929 20:27:23.305168 2870 leader_election.cc:288] T 422f3ab34bc047a1b3d38845a758377b P f2efee3001a740efaf8c26ef3cc7958a [CANDIDATE]: Term 1 election: Requested vote from peers d72848c751d44ffda4bb13aff2bc4b8b (127.17.53.66:34981), 2ae379b0bb064c688d7b22d223e956c5 (127.17.53.65:44837)
I20260929 20:27:23.305660 2664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "422f3ab34bc047a1b3d38845a758377b" candidate_uuid: "f2efee3001a740efaf8c26ef3cc7958a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d72848c751d44ffda4bb13aff2bc4b8b"
I20260929 20:27:23.305733 2530 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "422f3ab34bc047a1b3d38845a758377b" candidate_uuid: "f2efee3001a740efaf8c26ef3cc7958a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2ae379b0bb064c688d7b22d223e956c5"
I20260929 20:27:23.305893 2664 raft_consensus.cc:3037] T 422f3ab34bc047a1b3d38845a758377b P d72848c751d44ffda4bb13aff2bc4b8b [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:27:23.305975 2530 raft_consensus.cc:3037] T 422f3ab34bc047a1b3d38845a758377b P 2ae379b0bb064c688d7b22d223e956c5 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:27:23.309291 2530 raft_consensus.cc:2454] T 422f3ab34bc047a1b3d38845a758377b P 2ae379b0bb064c688d7b22d223e956c5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f2efee3001a740efaf8c26ef3cc7958a in term 1.
I20260929 20:27:23.309291 2664 raft_consensus.cc:2454] T 422f3ab34bc047a1b3d38845a758377b P d72848c751d44ffda4bb13aff2bc4b8b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f2efee3001a740efaf8c26ef3cc7958a in term 1.
I20260929 20:27:23.309784 2732 leader_election.cc:302] T 422f3ab34bc047a1b3d38845a758377b P f2efee3001a740efaf8c26ef3cc7958a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2ae379b0bb064c688d7b22d223e956c5, f2efee3001a740efaf8c26ef3cc7958a; no voters:
I20260929 20:27:23.310158 2870 raft_consensus.cc:2781] T 422f3ab34bc047a1b3d38845a758377b P f2efee3001a740efaf8c26ef3cc7958a [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:27:23.310616 2870 raft_consensus.cc:686] T 422f3ab34bc047a1b3d38845a758377b P f2efee3001a740efaf8c26ef3cc7958a [term 1 LEADER]: Becoming Leader. State: Replica: f2efee3001a740efaf8c26ef3cc7958a, State: Running, Role: LEADER
I20260929 20:27:23.311283 2870 consensus_queue.cc:237] T 422f3ab34bc047a1b3d38845a758377b P f2efee3001a740efaf8c26ef3cc7958a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d72848c751d44ffda4bb13aff2bc4b8b" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 34981 } } peers { permanent_uuid: "2ae379b0bb064c688d7b22d223e956c5" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 44837 } } peers { permanent_uuid: "f2efee3001a740efaf8c26ef3cc7958a" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 39253 } }
I20260929 20:27:23.316201 2386 catalog_manager.cc:5495] T 422f3ab34bc047a1b3d38845a758377b P f2efee3001a740efaf8c26ef3cc7958a reported cstate change: term changed from 0 to 1, leader changed from <none> to f2efee3001a740efaf8c26ef3cc7958a (127.17.53.67). New cstate: current_term: 1 leader_uuid: "f2efee3001a740efaf8c26ef3cc7958a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d72848c751d44ffda4bb13aff2bc4b8b" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 34981 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2ae379b0bb064c688d7b22d223e956c5" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 44837 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f2efee3001a740efaf8c26ef3cc7958a" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 39253 } health_report { overall_health: HEALTHY } } }
I20260929 20:27:23.367789 2860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:27:23.361649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45764 (local address 127.17.53.67:39253)
0929 20:27:23.361843 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0929 20:27:23.361849 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:27:23.361957 (+ 108us) server_negotiation.cc:407] Connection header received
0929 20:27:23.362149 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:27:23.362153 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:27:23.362241 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:27:23.362365 (+ 124us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 20:27:23.362835 (+ 470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:27:23.363598 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:27:23.365208 (+ 1610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:27:23.365605 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:27:23.365696 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:27:23.366401 (+ 705us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 20:27:23.367126 (+ 725us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 20:27:23.367251 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0929 20:27:23.367414 (+ 163us) server_negotiation.cc:299] Negotiation successful
0929 20:27:23.367485 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260929 20:27:23.376283 2664 raft_consensus.cc:1261] T 422f3ab34bc047a1b3d38845a758377b P d72848c751d44ffda4bb13aff2bc4b8b [term 1 FOLLOWER]: Refusing update from remote peer f2efee3001a740efaf8c26ef3cc7958a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:27:23.376286 2530 raft_consensus.cc:1261] T 422f3ab34bc047a1b3d38845a758377b P 2ae379b0bb064c688d7b22d223e956c5 [term 1 FOLLOWER]: Refusing update from remote peer f2efee3001a740efaf8c26ef3cc7958a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:27:23.377283 2872 consensus_queue.cc:1035] T 422f3ab34bc047a1b3d38845a758377b P f2efee3001a740efaf8c26ef3cc7958a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d72848c751d44ffda4bb13aff2bc4b8b" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 34981 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:27:23.377620 2870 consensus_queue.cc:1035] T 422f3ab34bc047a1b3d38845a758377b P f2efee3001a740efaf8c26ef3cc7958a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2ae379b0bb064c688d7b22d223e956c5" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 44837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:27:23.390126 2876 mvcc.cc:205] Tried to move back new op lower bound from 7334763083258486784 to 7334763083006955520. Current Snapshot: MvccSnapshot[applied={T|T < 7334763083258486784}]
I20260929 20:27:23.404256 2385 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50004:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 20:27:23.404580 2385 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50004:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 20:27:23.408569 2385 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 58314c300c934d2bbef023f492f2cafe: Sending DeleteTablet for 3 replicas of tablet 422f3ab34bc047a1b3d38845a758377b
I20260929 20:27:23.410123 2510 tablet_service.cc:1514] Processing DeleteTablet for tablet 422f3ab34bc047a1b3d38845a758377b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:27:23 UTC) from {username='kudu', principal='kudu/127.17.53.126@KRBTEST.COM'} at 127.0.0.1:34046
I20260929 20:27:23.410223 2644 tablet_service.cc:1514] Processing DeleteTablet for tablet 422f3ab34bc047a1b3d38845a758377b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:27:23 UTC) from {username='kudu', principal='kudu/127.17.53.126@KRBTEST.COM'} at 127.0.0.1:59256
I20260929 20:27:23.410818 2778 tablet_service.cc:1514] Processing DeleteTablet for tablet 422f3ab34bc047a1b3d38845a758377b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:27:23 UTC) from {username='kudu', principal='kudu/127.17.53.126@KRBTEST.COM'} at 127.0.0.1:45750
I20260929 20:27:23.413187 17621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu pid 2447
I20260929 20:27:23.425336 2885 tablet_replica.cc:330] T 422f3ab34bc047a1b3d38845a758377b P 2ae379b0bb064c688d7b22d223e956c5: stopping tablet replica
I20260929 20:27:23.425357 2886 tablet_replica.cc:330] T 422f3ab34bc047a1b3d38845a758377b P d72848c751d44ffda4bb13aff2bc4b8b: stopping tablet replica
I20260929 20:27:23.426054 2885 raft_consensus.cc:2229] T 422f3ab34bc047a1b3d38845a758377b P 2ae379b0bb064c688d7b22d223e956c5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 20:27:23.426064 2886 raft_consensus.cc:2229] T 422f3ab34bc047a1b3d38845a758377b P d72848c751d44ffda4bb13aff2bc4b8b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 20:27:23.426666 2886 raft_consensus.cc:2258] T 422f3ab34bc047a1b3d38845a758377b P d72848c751d44ffda4bb13aff2bc4b8b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:27:23.426658 2885 raft_consensus.cc:2258] T 422f3ab34bc047a1b3d38845a758377b P 2ae379b0bb064c688d7b22d223e956c5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:27:23.429265 2886 ts_tablet_manager.cc:1905] T 422f3ab34bc047a1b3d38845a758377b P d72848c751d44ffda4bb13aff2bc4b8b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:27:23.432335 2885 ts_tablet_manager.cc:1905] T 422f3ab34bc047a1b3d38845a758377b P 2ae379b0bb064c688d7b22d223e956c5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:27:23.438019 2886 ts_tablet_manager.cc:1918] T 422f3ab34bc047a1b3d38845a758377b P d72848c751d44ffda4bb13aff2bc4b8b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 20:27:23.438304 2886 log.cc:1196] T 422f3ab34bc047a1b3d38845a758377b P d72848c751d44ffda4bb13aff2bc4b8b: Deleting WAL directory at /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790713339161492-17621-0/minicluster-data/ts-1/wal/wals/422f3ab34bc047a1b3d38845a758377b
I20260929 20:27:23.439023 2886 ts_tablet_manager.cc:1939] T 422f3ab34bc047a1b3d38845a758377b P d72848c751d44ffda4bb13aff2bc4b8b: Deleting consensus metadata
I20260929 20:27:23.439596 2885 ts_tablet_manager.cc:1918] T 422f3ab34bc047a1b3d38845a758377b P 2ae379b0bb064c688d7b22d223e956c5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 20:27:23.439841 2885 log.cc:1196] T 422f3ab34bc047a1b3d38845a758377b P 2ae379b0bb064c688d7b22d223e956c5: Deleting WAL directory at /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790713339161492-17621-0/minicluster-data/ts-0/wal/wals/422f3ab34bc047a1b3d38845a758377b
I20260929 20:27:23.440586 2885 ts_tablet_manager.cc:1939] T 422f3ab34bc047a1b3d38845a758377b P 2ae379b0bb064c688d7b22d223e956c5: Deleting consensus metadata
I20260929 20:27:23.441069 2373 catalog_manager.cc:4841] TS d72848c751d44ffda4bb13aff2bc4b8b (127.17.53.66:34981): tablet 422f3ab34bc047a1b3d38845a758377b (table test-table [id=0c476a645cb4432b85c547e8b49aa58d]) successfully deleted
I20260929 20:27:23.442745 2371 catalog_manager.cc:4841] TS 2ae379b0bb064c688d7b22d223e956c5 (127.17.53.65:44837): tablet 422f3ab34bc047a1b3d38845a758377b (table test-table [id=0c476a645cb4432b85c547e8b49aa58d]) successfully deleted
I20260929 20:27:23.443502 2887 tablet_replica.cc:330] T 422f3ab34bc047a1b3d38845a758377b P f2efee3001a740efaf8c26ef3cc7958a: stopping tablet replica
I20260929 20:27:23.444029 2887 raft_consensus.cc:2229] T 422f3ab34bc047a1b3d38845a758377b P f2efee3001a740efaf8c26ef3cc7958a [term 1 LEADER]: Raft consensus shutting down.
I20260929 20:27:23.444823 2887 raft_consensus.cc:2258] T 422f3ab34bc047a1b3d38845a758377b P f2efee3001a740efaf8c26ef3cc7958a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:27:23.446842 2887 ts_tablet_manager.cc:1905] T 422f3ab34bc047a1b3d38845a758377b P f2efee3001a740efaf8c26ef3cc7958a: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 29 20:27:23 dist-test-slave-zbnc krb5kdc[2341](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713640, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.53.65@KRBTEST.COM
I20260929 20:27:23.454638 2887 ts_tablet_manager.cc:1918] T 422f3ab34bc047a1b3d38845a758377b P f2efee3001a740efaf8c26ef3cc7958a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 20:27:23.454821 2887 log.cc:1196] T 422f3ab34bc047a1b3d38845a758377b P f2efee3001a740efaf8c26ef3cc7958a: Deleting WAL directory at /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790713339161492-17621-0/minicluster-data/ts-2/wal/wals/422f3ab34bc047a1b3d38845a758377b
I20260929 20:27:23.455416 2887 ts_tablet_manager.cc:1939] T 422f3ab34bc047a1b3d38845a758377b P f2efee3001a740efaf8c26ef3cc7958a: Deleting consensus metadata
I20260929 20:27:23.456924 2859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:27:23.415704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34058 (local address 127.17.53.65:44837)
0929 20:27:23.415966 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0929 20:27:23.415975 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 20:27:23.440025 (+ 24050us) server_negotiation.cc:407] Connection header received
0929 20:27:23.440554 (+ 529us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:27:23.440560 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:27:23.440652 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:27:23.440724 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:27:23.442744 (+ 2020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:27:23.443739 (+ 995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:27:23.445107 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:27:23.445733 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:27:23.445889 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:27:23.450463 (+ 4574us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:27:23.450484 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:27:23.450498 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:27:23.450530 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:27:23.453743 (+ 3213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:27:23.454459 (+ 716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:27:23.454468 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:27:23.454472 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:27:23.454615 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:27:23.455177 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:27:23.455182 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:27:23.455185 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:27:23.455604 (+ 419us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:27:23.455913 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0929 20:27:23.456337 (+ 424us) server_negotiation.cc:299] Negotiation successful
0929 20:27:23.456555 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260929 20:27:23.457347 2371 catalog_manager.cc:4841] TS f2efee3001a740efaf8c26ef3cc7958a (127.17.53.67:39253): tablet 422f3ab34bc047a1b3d38845a758377b (table test-table [id=0c476a645cb4432b85c547e8b49aa58d]) successfully deleted
I20260929 20:27:23.457655 2570 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:27:23.604480 17621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu with pid 2447
I20260929 20:27:23.627990 17621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu pid 2581
Sep 29 20:27:23 dist-test-slave-zbnc krb5kdc[2341](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713640, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.53.66@KRBTEST.COM
I20260929 20:27:23.645615 2858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:27:23.628917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59260 (local address 127.17.53.66:34981)
0929 20:27:23.629105 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0929 20:27:23.629112 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:27:23.629131 (+ 19us) server_negotiation.cc:407] Connection header received
0929 20:27:23.629331 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:27:23.629338 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:27:23.629429 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:27:23.629562 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:27:23.631073 (+ 1511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:27:23.632196 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:27:23.633556 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:27:23.634189 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:27:23.634320 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:27:23.638752 (+ 4432us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:27:23.638771 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:27:23.638794 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:27:23.638845 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:27:23.642594 (+ 3749us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:27:23.643461 (+ 867us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:27:23.643471 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:27:23.643474 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:27:23.643624 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:27:23.644037 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:27:23.644043 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:27:23.644046 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:27:23.644531 (+ 485us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:27:23.644830 (+ 299us) server_negotiation.cc:1035] Waiting for connection context
0929 20:27:23.645020 (+ 190us) server_negotiation.cc:299] Negotiation successful
0929 20:27:23.645240 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 20:27:23.646399 2704 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:27:23.737568 17621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu with pid 2581
I20260929 20:27:23.755244 17621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu pid 2715
Sep 29 20:27:23 dist-test-slave-zbnc krb5kdc[2341](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713640, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.53.67@KRBTEST.COM
I20260929 20:27:23.773923 2860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:27:23.756065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45766 (local address 127.17.53.67:39253)
0929 20:27:23.756269 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0929 20:27:23.756275 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:27:23.756297 (+ 22us) server_negotiation.cc:407] Connection header received
0929 20:27:23.756440 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:27:23.756446 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:27:23.756548 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:27:23.756678 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:27:23.758010 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:27:23.759233 (+ 1223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:27:23.760521 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:27:23.761226 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:27:23.761373 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:27:23.766609 (+ 5236us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:27:23.766624 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:27:23.766646 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:27:23.766697 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:27:23.770900 (+ 4203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:27:23.771656 (+ 756us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:27:23.771666 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:27:23.771669 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:27:23.771878 (+ 209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:27:23.772238 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:27:23.772243 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:27:23.772247 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:27:23.772727 (+ 480us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:27:23.773077 (+ 350us) server_negotiation.cc:1035] Waiting for connection context
0929 20:27:23.773258 (+ 181us) server_negotiation.cc:299] Negotiation successful
0929 20:27:23.773489 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260929 20:27:23.774667 2838 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:27:23.898556 17621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu with pid 2715
I20260929 20:27:23.919085 17621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu pid 2354
I20260929 20:27:23.920385 2416 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:27:24.030934 17621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu with pid 2354
2026-09-29T20:27:24Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (4738 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-29T20:27:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T20:27:24Z Disabled control of system clock
I20260929 20:27:24.109702 17621 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
/tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790713339161492-17621-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790713339161492-17621-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790713339161492-17621-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790713339161492-17621-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.53.126:43737
--webserver_interface=127.17.53.126
--webserver_port=0
--builtin_ntp_servers=127.17.53.84:45257
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.53.126:43737
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 20:27:24.453395 2896 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:27:24.453706 2896 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:27:24.453845 2896 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:27:24.461215 2896 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:27:24.461323 2896 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:27:24.461400 2896 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:27:24.461524 2896 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:27:24.461593 2896 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:27:24.470849 2896 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.53.84:45257
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790713339161492-17621-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790713339161492-17621-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.53.126:43737
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.53.126:43737
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790713339161492-17621-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.53.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790713339161492-17621-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:27:24.472270 2896 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:27:24.474108 2896 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:27:24.486523 2902 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:27:24.486541 2904 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:27:24.487084 2901 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:27:24.487473 2896 server_base.cc:1034] running on GCE node
I20260929 20:27:24.488209 2896 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:27:24.489665 2896 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:27:24.490979 2896 hybrid_clock.cc:648] HybridClock initialized: now 1790713644490905 us; error 50 us; skew 500 ppm
I20260929 20:27:24.491304 2896 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 20:27:24.502074 2896 webserver.cc:460] Webserver started at http://127.17.53.126:46109/ using document root <none> and password file <none>
I20260929 20:27:24.502777 2896 fs_manager.cc:359] Metadata directory not provided
I20260929 20:27:24.502954 2896 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:27:24.503299 2896 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:27:24.506291 2896 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790713339161492-17621-0/minicluster-data/master-0/data/instance:
uuid: "3a7c2d3af7964f57a509d9f479725278"
format_stamp: "Formatted at 2026-09-29 20:27:24 on dist-test-slave-zbnc"
server_key: "c0ab9ea2d4f84d806ed658acccc8a0c8"
server_key_iv: "a352b05da4affea823ffda33a9b1d22a"
server_key_version: "encryptionkey@0"
I20260929 20:27:24.507174 2896 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790713339161492-17621-0/minicluster-data/master-0/wal/instance:
uuid: "3a7c2d3af7964f57a509d9f479725278"
format_stamp: "Formatted at 2026-09-29 20:27:24 on dist-test-slave-zbnc"
server_key: "c0ab9ea2d4f84d806ed658acccc8a0c8"
server_key_iv: "a352b05da4affea823ffda33a9b1d22a"
server_key_version: "encryptionkey@0"
I20260929 20:27:24.512768 2896 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 20:27:24.516503 2910 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:27:24.518064 2896 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.004s
I20260929 20:27:24.518216 2896 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790713339161492-17621-0/minicluster-data/master-0/data,/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790713339161492-17621-0/minicluster-data/master-0/wal
uuid: "3a7c2d3af7964f57a509d9f479725278"
format_stamp: "Formatted at 2026-09-29 20:27:24 on dist-test-slave-zbnc"
server_key: "c0ab9ea2d4f84d806ed658acccc8a0c8"
server_key_iv: "a352b05da4affea823ffda33a9b1d22a"
server_key_version: "encryptionkey@0"
I20260929 20:27:24.518409 2896 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790713339161492-17621-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790713339161492-17621-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790713339161492-17621-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:27:24.536456 2896 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:27:24.537272 2896 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:27:24.537514 2896 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:27:24.556656 2896 rpc_server.cc:307] RPC server started. Bound to: 127.17.53.126:43737
I20260929 20:27:24.556681 2961 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.53.126:43737 every 8 connection(s)
I20260929 20:27:24.558060 2896 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790713339161492-17621-0/minicluster-data/master-0/data/info.pb
I20260929 20:27:24.561995 2962 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:27:24.563974 17621 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu as pid 2896
I20260929 20:27:24.564162 17621 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790713339161492-17621-0/minicluster-data/master-0/wal/instance
I20260929 20:27:24.564724 17621 external_mini_cluster.cc:1428] Setting key ea81b488fed267aa44fc7286e6e28ae2
I20260929 20:27:24.575441 2962 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3a7c2d3af7964f57a509d9f479725278: Bootstrap starting.
I20260929 20:27:24.578102 2965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:27:24.566803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45154 (local address 127.17.53.126:43737)
0929 20:27:24.567879 (+ 1076us) server_negotiation.cc:206] Beginning negotiation
0929 20:27:24.567898 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0929 20:27:24.567939 (+ 41us) server_negotiation.cc:407] Connection header received
0929 20:27:24.569604 (+ 1665us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:27:24.569627 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:27:24.570096 (+ 469us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:27:24.570510 (+ 414us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:27:24.571107 (+ 597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:27:24.572365 (+ 1258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:27:24.573753 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:27:24.574418 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:27:24.574607 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:27:24.575850 (+ 1243us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:27:24.575858 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:27:24.575871 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:27:24.576391 (+ 520us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:27:24.576486 (+ 95us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:27:24.576507 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:27:24.576631 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0929 20:27:24.576877 (+ 246us) server_negotiation.cc:299] Negotiation successful
0929 20:27:24.577166 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":656,"thread_start_us":390,"threads_started":1}
I20260929 20:27:24.578751 2962 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3a7c2d3af7964f57a509d9f479725278: Neither blocks nor log segments found. Creating new log.
I20260929 20:27:24.579942 2962 log.cc:824] T 00000000000000000000000000000000 P 3a7c2d3af7964f57a509d9f479725278: Log is configured to *not* fsync() on all Append() calls
I20260929 20:27:24.582708 2962 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3a7c2d3af7964f57a509d9f479725278: No bootstrap required, opened a new log
I20260929 20:27:24.590214 2962 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3a7c2d3af7964f57a509d9f479725278 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a7c2d3af7964f57a509d9f479725278" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 43737 } }
I20260929 20:27:24.590636 2962 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3a7c2d3af7964f57a509d9f479725278 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:27:24.590752 2962 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3a7c2d3af7964f57a509d9f479725278 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a7c2d3af7964f57a509d9f479725278, State: Initialized, Role: FOLLOWER
I20260929 20:27:24.591346 2962 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3a7c2d3af7964f57a509d9f479725278 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a7c2d3af7964f57a509d9f479725278" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 43737 } }
I20260929 20:27:24.591555 2962 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3a7c2d3af7964f57a509d9f479725278 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 20:27:24.591652 2962 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3a7c2d3af7964f57a509d9f479725278 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 20:27:24.591773 2962 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3a7c2d3af7964f57a509d9f479725278 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:27:24.593957 2962 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3a7c2d3af7964f57a509d9f479725278 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a7c2d3af7964f57a509d9f479725278" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 43737 } }
I20260929 20:27:24.594543 2962 leader_election.cc:302] T 00000000000000000000000000000000 P 3a7c2d3af7964f57a509d9f479725278 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3a7c2d3af7964f57a509d9f479725278; no voters:
I20260929 20:27:24.595252 2962 leader_election.cc:288] T 00000000000000000000000000000000 P 3a7c2d3af7964f57a509d9f479725278 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 20:27:24.595412 2967 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3a7c2d3af7964f57a509d9f479725278 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:27:24.596158 2967 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3a7c2d3af7964f57a509d9f479725278 [term 1 LEADER]: Becoming Leader. State: Replica: 3a7c2d3af7964f57a509d9f479725278, State: Running, Role: LEADER
I20260929 20:27:24.596629 2967 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3a7c2d3af7964f57a509d9f479725278 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a7c2d3af7964f57a509d9f479725278" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 43737 } }
I20260929 20:27:24.597383 2962 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3a7c2d3af7964f57a509d9f479725278 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:27:24.599355 2969 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a7c2d3af7964f57a509d9f479725278 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3a7c2d3af7964f57a509d9f479725278. Latest consensus state: current_term: 1 leader_uuid: "3a7c2d3af7964f57a509d9f479725278" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a7c2d3af7964f57a509d9f479725278" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 43737 } } }
I20260929 20:27:24.599413 2968 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a7c2d3af7964f57a509d9f479725278 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3a7c2d3af7964f57a509d9f479725278" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a7c2d3af7964f57a509d9f479725278" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 43737 } } }
I20260929 20:27:24.600067 2968 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a7c2d3af7964f57a509d9f479725278 [sys.catalog]: This master's current role is: LEADER
I20260929 20:27:24.599941 2969 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a7c2d3af7964f57a509d9f479725278 [sys.catalog]: This master's current role is: LEADER
I20260929 20:27:24.602176 2973 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:27:24.608438 2973 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:27:24.617156 2973 catalog_manager.cc:1347] Generated new cluster ID: 0e4e441f160140aea82b56ec6feca608
I20260929 20:27:24.617275 2973 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:27:24.643967 2973 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 20:27:24.645622 2973 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:27:24.654384 2973 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3a7c2d3af7964f57a509d9f479725278: Generated new TSK 0
I20260929 20:27:24.655347 2973 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 20:27:24.673188 17621 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
/tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790713339161492-17621-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790713339161492-17621-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790713339161492-17621-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790713339161492-17621-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.53.65:0
--local_ip_for_outbound_sockets=127.17.53.65
--webserver_interface=127.17.53.65
--webserver_port=0
--tserver_master_addrs=127.17.53.126:43737
--builtin_ntp_servers=127.17.53.84:45257
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 20:27:25.004739 2986 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:27:25.005077 2986 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:27:25.005244 2986 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:27:25.012965 2986 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:27:25.013098 2986 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:27:25.013307 2986 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.53.65
I20260929 20:27:25.023015 2986 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.53.84:45257
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790713339161492-17621-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790713339161492-17621-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.53.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790713339161492-17621-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.17.53.65
--webserver_port=0
--tserver_master_addrs=127.17.53.126:43737
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.53.65
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790713339161492-17621-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:27:25.024351 2986 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:27:25.026182 2986 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:27:25.039767 2991 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:27:25.039769 2992 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:27:25.039748 2994 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:27:25.044667 2986 server_base.cc:1034] running on GCE node
I20260929 20:27:25.045342 2986 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:27:25.046341 2986 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:27:25.047549 2986 hybrid_clock.cc:648] HybridClock initialized: now 1790713645047503 us; error 32 us; skew 500 ppm
I20260929 20:27:25.047837 2986 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 20:27:25.050276 2986 webserver.cc:460] Webserver started at http://127.17.53.65:37249/ using document root <none> and password file <none>
I20260929 20:27:25.051095 2986 fs_manager.cc:359] Metadata directory not provided
I20260929 20:27:25.051184 2986 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:27:25.051489 2986 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:27:25.054401 2986 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790713339161492-17621-0/minicluster-data/ts-0/data/instance:
uuid: "a939b910230949ed8f99749dd8b8c84f"
format_stamp: "Formatted at 2026-09-29 20:27:25 on dist-test-slave-zbnc"
server_key: "b3a9bed6acb9df2e20682945a2830baa"
server_key_iv: "af6d7bbfd4a3204b3964ca3bcc517085"
server_key_version: "encryptionkey@0"
I20260929 20:27:25.055320 2986 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790713339161492-17621-0/minicluster-data/ts-0/wal/instance:
uuid: "a939b910230949ed8f99749dd8b8c84f"
format_stamp: "Formatted at 2026-09-29 20:27:25 on dist-test-slave-zbnc"
server_key: "b3a9bed6acb9df2e20682945a2830baa"
server_key_iv: "af6d7bbfd4a3204b3964ca3bcc517085"
server_key_version: "encryptionkey@0"
I20260929 20:27:25.062346 2986 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.000s
I20260929 20:27:25.065809 3000 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:27:25.067303 2986 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260929 20:27:25.067447 2986 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790713339161492-17621-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790713339161492-17621-0/minicluster-data/ts-0/wal
uuid: "a939b910230949ed8f99749dd8b8c84f"
format_stamp: "Formatted at 2026-09-29 20:27:25 on dist-test-slave-zbnc"
server_key: "b3a9bed6acb9df2e20682945a2830baa"
server_key_iv: "af6d7bbfd4a3204b3964ca3bcc517085"
server_key_version: "encryptionkey@0"
I20260929 20:27:25.067633 2986 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790713339161492-17621-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790713339161492-17621-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790713339161492-17621-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:27:25.087411 2986 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:27:25.088202 2986 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:27:25.088469 2986 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:27:25.089546 2986 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:27:25.091076 2986 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:27:25.091169 2986 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:27:25.091248 2986 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:27:25.091305 2986 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:27:25.125082 2986 rpc_server.cc:307] RPC server started. Bound to: 127.17.53.65:42299
I20260929 20:27:25.125115 3113 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.53.65:42299 every 8 connection(s)
I20260929 20:27:25.126400 2986 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790713339161492-17621-0/minicluster-data/ts-0/data/info.pb
I20260929 20:27:25.128002 17621 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu as pid 2986
I20260929 20:27:25.128197 17621 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790713339161492-17621-0/minicluster-data/ts-0/wal/instance
I20260929 20:27:25.128736 17621 external_mini_cluster.cc:1428] Setting key 998394fc8693f5040a42036f88a92180
I20260929 20:27:25.131924 17621 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
/tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790713339161492-17621-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790713339161492-17621-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790713339161492-17621-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790713339161492-17621-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.53.66:0
--local_ip_for_outbound_sockets=127.17.53.66
--webserver_interface=127.17.53.66
--webserver_port=0
--tserver_master_addrs=127.17.53.126:43737
--builtin_ntp_servers=127.17.53.84:45257
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 20:27:25.141038 3116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:27:25.129771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.53.65:40687 (local address 127.17.53.126:43737)
0929 20:27:25.130251 (+ 480us) server_negotiation.cc:206] Beginning negotiation
0929 20:27:25.130257 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:27:25.131850 (+ 1593us) server_negotiation.cc:407] Connection header received
0929 20:27:25.133776 (+ 1926us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:27:25.133781 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:27:25.133871 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:27:25.134004 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:27:25.135187 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:27:25.135989 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:27:25.137618 (+ 1629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:27:25.138058 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:27:25.138199 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:27:25.139934 (+ 1735us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:27:25.139943 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:27:25.139951 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:27:25.139979 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:27:25.140034 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:27:25.140053 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:27:25.140148 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0929 20:27:25.140462 (+ 314us) server_negotiation.cc:299] Negotiation successful
0929 20:27:25.140585 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":364,"thread_start_us":254,"threads_started":1}
I20260929 20:27:25.142056 3117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:27:25.130464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:43737 (local address 127.17.53.65:40687)
0929 20:27:25.131681 (+ 1217us) negotiation.cc:107] Waiting for socket to connect
0929 20:27:25.131732 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0929 20:27:25.133495 (+ 1763us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:27:25.134250 (+ 755us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:27:25.134259 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:27:25.134356 (+ 97us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:27:25.135020 (+ 664us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:27:25.135039 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:27:25.136161 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:27:25.136166 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:27:25.137465 (+ 1299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:27:25.137478 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:27:25.138298 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:27:25.138308 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:27:25.138652 (+ 344us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:27:25.139514 (+ 862us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 20:27:25.139519 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:27:25.139580 (+ 61us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 20:27:25.139703 (+ 123us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:27:25.140206 (+ 503us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:27:25.140215 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:27:25.140223 (+ 8us) client_negotiation.cc:769] Sending connection context
0929 20:27:25.140403 (+ 180us) client_negotiation.cc:240] Negotiation successful
0929 20:27:25.140758 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":666,"thread_start_us":419,"threads_started":1}
I20260929 20:27:25.143971 3114 heartbeater.cc:347] Connected to a master server at 127.17.53.126:43737
I20260929 20:27:25.144307 3114 heartbeater.cc:464] Registering TS with master...
I20260929 20:27:25.145153 3114 heartbeater.cc:511] Master 127.17.53.126:43737 requested a full tablet report, sending...
I20260929 20:27:25.147630 2927 ts_manager.cc:194] Registered new tserver with Master: a939b910230949ed8f99749dd8b8c84f (127.17.53.65:42299)
I20260929 20:27:25.149509 2927 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.17.53.65:40687
W20260929 20:27:25.470151 3118 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:27:25.470458 3118 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:27:25.470646 3118 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:27:25.478659 3118 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:27:25.478760 3118 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:27:25.478986 3118 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.53.66
I20260929 20:27:25.489084 3118 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.53.84:45257
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790713339161492-17621-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790713339161492-17621-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.53.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790713339161492-17621-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.17.53.66
--webserver_port=0
--tserver_master_addrs=127.17.53.126:43737
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.53.66
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790713339161492-17621-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:27:25.490474 3118 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:27:25.492429 3118 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:27:25.507659 3124 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:27:25.507684 3123 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:27:25.508476 3126 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:27:25.508507 3118 server_base.cc:1034] running on GCE node
I20260929 20:27:25.509145 3118 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:27:25.510128 3118 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:27:25.511358 3118 hybrid_clock.cc:648] HybridClock initialized: now 1790713645511320 us; error 34 us; skew 500 ppm
I20260929 20:27:25.511698 3118 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 20:27:25.514227 3118 webserver.cc:460] Webserver started at http://127.17.53.66:42279/ using document root <none> and password file <none>
I20260929 20:27:25.514987 3118 fs_manager.cc:359] Metadata directory not provided
I20260929 20:27:25.515120 3118 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:27:25.515442 3118 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:27:25.518322 3118 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790713339161492-17621-0/minicluster-data/ts-1/data/instance:
uuid: "5f877d75d3b345bab8d9a9981ea9690c"
format_stamp: "Formatted at 2026-09-29 20:27:25 on dist-test-slave-zbnc"
server_key: "ba131e425f4b82cd15e39661c290e646"
server_key_iv: "bfa45e14472513502abef50dfd98205f"
server_key_version: "encryptionkey@0"
I20260929 20:27:25.519171 3118 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790713339161492-17621-0/minicluster-data/ts-1/wal/instance:
uuid: "5f877d75d3b345bab8d9a9981ea9690c"
format_stamp: "Formatted at 2026-09-29 20:27:25 on dist-test-slave-zbnc"
server_key: "ba131e425f4b82cd15e39661c290e646"
server_key_iv: "bfa45e14472513502abef50dfd98205f"
server_key_version: "encryptionkey@0"
I20260929 20:27:25.524710 3118 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 20:27:25.528448 3132 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:27:25.529843 3118 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 20:27:25.530066 3118 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790713339161492-17621-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790713339161492-17621-0/minicluster-data/ts-1/wal
uuid: "5f877d75d3b345bab8d9a9981ea9690c"
format_stamp: "Formatted at 2026-09-29 20:27:25 on dist-test-slave-zbnc"
server_key: "ba131e425f4b82cd15e39661c290e646"
server_key_iv: "bfa45e14472513502abef50dfd98205f"
server_key_version: "encryptionkey@0"
I20260929 20:27:25.530290 3118 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790713339161492-17621-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790713339161492-17621-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790713339161492-17621-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:27:25.560628 3118 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:27:25.561497 3118 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:27:25.561810 3118 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:27:25.562832 3118 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:27:25.564388 3118 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:27:25.564508 3118 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:27:25.564638 3118 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:27:25.564723 3118 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:27:25.601876 3118 rpc_server.cc:307] RPC server started. Bound to: 127.17.53.66:40629
I20260929 20:27:25.601989 3245 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.53.66:40629 every 8 connection(s)
I20260929 20:27:25.603384 3118 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790713339161492-17621-0/minicluster-data/ts-1/data/info.pb
I20260929 20:27:25.609767 17621 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu as pid 3118
I20260929 20:27:25.609972 17621 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790713339161492-17621-0/minicluster-data/ts-1/wal/instance
I20260929 20:27:25.610638 17621 external_mini_cluster.cc:1428] Setting key 903934687561a8e73fc9bc4be8bacc6c
I20260929 20:27:25.613812 17621 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
/tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790713339161492-17621-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790713339161492-17621-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790713339161492-17621-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790713339161492-17621-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.53.67:0
--local_ip_for_outbound_sockets=127.17.53.67
--webserver_interface=127.17.53.67
--webserver_port=0
--tserver_master_addrs=127.17.53.126:43737
--builtin_ntp_servers=127.17.53.84:45257
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 20:27:25.618108 3116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:27:25.606380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.53.66:53509 (local address 127.17.53.126:43737)
0929 20:27:25.606592 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0929 20:27:25.606602 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 20:27:25.608163 (+ 1561us) server_negotiation.cc:407] Connection header received
0929 20:27:25.610478 (+ 2315us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:27:25.610482 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:27:25.610550 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:27:25.610638 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:27:25.612213 (+ 1575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:27:25.613032 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:27:25.615011 (+ 1979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:27:25.615463 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:27:25.615618 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:27:25.617215 (+ 1597us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:27:25.617221 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:27:25.617224 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:27:25.617244 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:27:25.617277 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:27:25.617308 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:27:25.617415 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0929 20:27:25.617662 (+ 247us) server_negotiation.cc:299] Negotiation successful
0929 20:27:25.617769 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260929 20:27:25.619360 3248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:27:25.606979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:43737 (local address 127.17.53.66:53509)
0929 20:27:25.607996 (+ 1017us) negotiation.cc:107] Waiting for socket to connect
0929 20:27:25.608037 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0929 20:27:25.610132 (+ 2095us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:27:25.611020 (+ 888us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:27:25.611034 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:27:25.611173 (+ 139us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:27:25.612030 (+ 857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:27:25.612051 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:27:25.613203 (+ 1152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:27:25.613209 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:27:25.614800 (+ 1591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:27:25.614814 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:27:25.615713 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:27:25.615721 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:27:25.615983 (+ 262us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:27:25.616752 (+ 769us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 20:27:25.616754 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:27:25.616896 (+ 142us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 20:27:25.616987 (+ 91us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:27:25.617413 (+ 426us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:27:25.617427 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:27:25.617438 (+ 11us) client_negotiation.cc:769] Sending connection context
0929 20:27:25.617636 (+ 198us) client_negotiation.cc:240] Negotiation successful
0929 20:27:25.618094 (+ 458us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":491,"thread_start_us":326,"threads_started":1}
I20260929 20:27:25.621129 3246 heartbeater.cc:347] Connected to a master server at 127.17.53.126:43737
I20260929 20:27:25.621479 3246 heartbeater.cc:464] Registering TS with master...
I20260929 20:27:25.622239 3246 heartbeater.cc:511] Master 127.17.53.126:43737 requested a full tablet report, sending...
I20260929 20:27:25.623865 2927 ts_manager.cc:194] Registered new tserver with Master: 5f877d75d3b345bab8d9a9981ea9690c (127.17.53.66:40629)
I20260929 20:27:25.624783 2927 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.17.53.66:53509
W20260929 20:27:25.942334 3249 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:27:25.942656 3249 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:27:25.942818 3249 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:27:25.950093 3249 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:27:25.950222 3249 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:27:25.950403 3249 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.53.67
I20260929 20:27:25.959648 3249 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.53.84:45257
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790713339161492-17621-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790713339161492-17621-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.53.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790713339161492-17621-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.17.53.67
--webserver_port=0
--tserver_master_addrs=127.17.53.126:43737
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.53.67
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790713339161492-17621-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:27:25.961005 3249 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:27:25.962777 3249 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:27:25.976302 3254 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:27:25.976341 3257 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:27:25.976399 3255 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:27:25.977069 3249 server_base.cc:1034] running on GCE node
I20260929 20:27:25.977805 3249 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:27:25.978796 3249 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:27:25.980044 3249 hybrid_clock.cc:648] HybridClock initialized: now 1790713645979998 us; error 43 us; skew 500 ppm
I20260929 20:27:25.980417 3249 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 20:27:25.982647 3249 webserver.cc:460] Webserver started at http://127.17.53.67:39847/ using document root <none> and password file <none>
I20260929 20:27:25.983415 3249 fs_manager.cc:359] Metadata directory not provided
I20260929 20:27:25.983538 3249 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:27:25.983855 3249 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:27:25.986644 3249 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790713339161492-17621-0/minicluster-data/ts-2/data/instance:
uuid: "04e06a44c1b64960b4f41240d5fc7f16"
format_stamp: "Formatted at 2026-09-29 20:27:25 on dist-test-slave-zbnc"
server_key: "efe9ef767152a4d52911283305fd8ec6"
server_key_iv: "d2de260d9abf75013d9c92818df62878"
server_key_version: "encryptionkey@0"
I20260929 20:27:25.987480 3249 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790713339161492-17621-0/minicluster-data/ts-2/wal/instance:
uuid: "04e06a44c1b64960b4f41240d5fc7f16"
format_stamp: "Formatted at 2026-09-29 20:27:25 on dist-test-slave-zbnc"
server_key: "efe9ef767152a4d52911283305fd8ec6"
server_key_iv: "d2de260d9abf75013d9c92818df62878"
server_key_version: "encryptionkey@0"
I20260929 20:27:25.992522 3249 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260929 20:27:25.996006 3263 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:27:25.997355 3249 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 20:27:25.997555 3249 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790713339161492-17621-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790713339161492-17621-0/minicluster-data/ts-2/wal
uuid: "04e06a44c1b64960b4f41240d5fc7f16"
format_stamp: "Formatted at 2026-09-29 20:27:25 on dist-test-slave-zbnc"
server_key: "efe9ef767152a4d52911283305fd8ec6"
server_key_iv: "d2de260d9abf75013d9c92818df62878"
server_key_version: "encryptionkey@0"
I20260929 20:27:25.997740 3249 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790713339161492-17621-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790713339161492-17621-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790713339161492-17621-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:27:26.010711 3249 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:27:26.011646 3249 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:27:26.011905 3249 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:27:26.012835 3249 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:27:26.014324 3249 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:27:26.014418 3249 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:27:26.014493 3249 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:27:26.014555 3249 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:27:26.050355 3249 rpc_server.cc:307] RPC server started. Bound to: 127.17.53.67:37037
I20260929 20:27:26.050477 3375 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.53.67:37037 every 8 connection(s)
I20260929 20:27:26.051786 3249 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790713339161492-17621-0/minicluster-data/ts-2/data/info.pb
I20260929 20:27:26.060732 17621 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu as pid 3249
I20260929 20:27:26.060917 17621 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790713339161492-17621-0/minicluster-data/ts-2/wal/instance
I20260929 20:27:26.061405 17621 external_mini_cluster.cc:1428] Setting key c5c3c55c5b788eff033b02192fd7a4ec
I20260929 20:27:26.065318 3116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:27:26.054444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.53.67:34251 (local address 127.17.53.126:43737)
0929 20:27:26.054631 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0929 20:27:26.054636 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:27:26.056149 (+ 1513us) server_negotiation.cc:407] Connection header received
0929 20:27:26.058042 (+ 1893us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:27:26.058048 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:27:26.058146 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:27:26.058272 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:27:26.059452 (+ 1180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:27:26.060230 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:27:26.062090 (+ 1860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:27:26.062715 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:27:26.062835 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:27:26.064533 (+ 1698us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:27:26.064539 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:27:26.064544 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:27:26.064565 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:27:26.064607 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:27:26.064622 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:27:26.064720 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0929 20:27:26.065008 (+ 288us) server_negotiation.cc:299] Negotiation successful
0929 20:27:26.065086 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 20:27:26.066293 3378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:27:26.054965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:43737 (local address 127.17.53.67:34251)
0929 20:27:26.055960 (+ 995us) negotiation.cc:107] Waiting for socket to connect
0929 20:27:26.056006 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0929 20:27:26.057743 (+ 1737us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:27:26.058512 (+ 769us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:27:26.058521 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:27:26.058607 (+ 86us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:27:26.059262 (+ 655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:27:26.059280 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:27:26.060403 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:27:26.060416 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:27:26.061924 (+ 1508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:27:26.061937 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:27:26.062994 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:27:26.063004 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:27:26.063336 (+ 332us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:27:26.064163 (+ 827us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 20:27:26.064166 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:27:26.064206 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 20:27:26.064312 (+ 106us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:27:26.064757 (+ 445us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:27:26.064769 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:27:26.064777 (+ 8us) client_negotiation.cc:769] Sending connection context
0929 20:27:26.064981 (+ 204us) client_negotiation.cc:240] Negotiation successful
0929 20:27:26.065264 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":277,"threads_started":1}
I20260929 20:27:26.067615 3376 heartbeater.cc:347] Connected to a master server at 127.17.53.126:43737
I20260929 20:27:26.067951 3376 heartbeater.cc:464] Registering TS with master...
I20260929 20:27:26.068647 3376 heartbeater.cc:511] Master 127.17.53.126:43737 requested a full tablet report, sending...
I20260929 20:27:26.070014 2927 ts_manager.cc:194] Registered new tserver with Master: 04e06a44c1b64960b4f41240d5fc7f16 (127.17.53.67:37037)
I20260929 20:27:26.070787 2927 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.17.53.67:34251
I20260929 20:27:26.077013 17621 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 20:27:26.086683 3116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:27:26.081036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45170 (local address 127.17.53.126:43737)
0929 20:27:26.081234 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0929 20:27:26.081238 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:27:26.081638 (+ 400us) server_negotiation.cc:407] Connection header received
0929 20:27:26.081775 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:27:26.081778 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:27:26.081845 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:27:26.081926 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:27:26.082438 (+ 512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:27:26.083188 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:27:26.084722 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:27:26.085112 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:27:26.085199 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:27:26.086119 (+ 920us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:27:26.086124 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:27:26.086127 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:27:26.086143 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:27:26.086174 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:27:26.086185 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:27:26.086252 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0929 20:27:26.086396 (+ 144us) server_negotiation.cc:299] Negotiation successful
0929 20:27:26.086468 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 20:27:26.090334 2927 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:45170:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 20:27:26.097580 2927 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 20:27:26.137598 3390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:27:26.111588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52864 (local address 127.17.53.66:40629)
0929 20:27:26.126603 (+ 15015us) server_negotiation.cc:206] Beginning negotiation
0929 20:27:26.126623 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0929 20:27:26.128691 (+ 2068us) server_negotiation.cc:407] Connection header received
0929 20:27:26.128803 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:27:26.128811 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:27:26.129148 (+ 337us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:27:26.129344 (+ 196us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:27:26.130133 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:27:26.131704 (+ 1571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:27:26.134674 (+ 2970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:27:26.136256 (+ 1582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:27:26.136415 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:27:26.136701 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0929 20:27:26.137054 (+ 353us) server_negotiation.cc:299] Negotiation successful
0929 20:27:26.137240 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14809,"thread_start_us":15487,"threads_started":1}
I20260929 20:27:26.137633 3388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:27:26.111343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.67:37037 (local address 127.0.0.1:55694)
0929 20:27:26.126839 (+ 15496us) negotiation.cc:107] Waiting for socket to connect
0929 20:27:26.126925 (+ 86us) client_negotiation.cc:174] Beginning negotiation
0929 20:27:26.127167 (+ 242us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:27:26.128217 (+ 1050us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:27:26.128228 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:27:26.128270 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:27:26.128734 (+ 464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:27:26.128754 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:27:26.131588 (+ 2834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:27:26.131597 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:27:26.134254 (+ 2657us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:27:26.134271 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:27:26.136713 (+ 2442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:27:26.136719 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:27:26.136892 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:27:26.136925 (+ 33us) client_negotiation.cc:769] Sending connection context
0929 20:27:26.136979 (+ 54us) client_negotiation.cc:240] Negotiation successful
0929 20:27:26.137205 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15150,"mutex_wait_us":146,"thread_start_us":15934,"threads_started":1}
I20260929 20:27:26.138602 3391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:27:26.111605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55694 (local address 127.17.53.67:37037)
0929 20:27:26.127174 (+ 15569us) server_negotiation.cc:206] Beginning negotiation
0929 20:27:26.127182 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 20:27:26.127222 (+ 40us) server_negotiation.cc:407] Connection header received
0929 20:27:26.127559 (+ 337us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:27:26.127571 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:27:26.127934 (+ 363us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:27:26.128983 (+ 1049us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:27:26.129282 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:27:26.131388 (+ 2106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:27:26.134674 (+ 3286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:27:26.136489 (+ 1815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:27:26.137141 (+ 652us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:27:26.137417 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0929 20:27:26.137488 (+ 71us) server_negotiation.cc:299] Negotiation successful
0929 20:27:26.137669 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15384,"thread_start_us":15208,"threads_started":1}
I20260929 20:27:26.138623 3389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:27:26.111343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.66:40629 (local address 127.0.0.1:52864)
0929 20:27:26.128382 (+ 17039us) negotiation.cc:107] Waiting for socket to connect
0929 20:27:26.128406 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0929 20:27:26.128552 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:27:26.129589 (+ 1037us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:27:26.129594 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:27:26.129619 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:27:26.129962 (+ 343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:27:26.129977 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:27:26.131875 (+ 1898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:27:26.131882 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:27:26.134274 (+ 2392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:27:26.134290 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:27:26.136485 (+ 2195us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:27:26.136494 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:27:26.136781 (+ 287us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:27:26.136841 (+ 60us) client_negotiation.cc:769] Sending connection context
0929 20:27:26.136977 (+ 136us) client_negotiation.cc:240] Negotiation successful
0929 20:27:26.138206 (+ 1229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16875,"mutex_wait_us":315,"thread_start_us":16672,"threads_started":1}
I20260929 20:27:26.142442 3181 tablet_service.cc:1467] Processing CreateTablet for tablet fc1c9c0317ff48939844703bb04b1d2c (DEFAULT_TABLE table=test-table [id=6c29b3be2a3b4659a5c88867099c63be]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:27:26.143286 3393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:27:26.134235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.65:42299 (local address 127.0.0.1:39292)
0929 20:27:26.134981 (+ 746us) negotiation.cc:107] Waiting for socket to connect
0929 20:27:26.135003 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 20:27:26.135131 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:27:26.136038 (+ 907us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:27:26.136043 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:27:26.136069 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:27:26.136533 (+ 464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:27:26.136547 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:27:26.138480 (+ 1933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:27:26.138484 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:27:26.140614 (+ 2130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:27:26.140631 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:27:26.142475 (+ 1844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:27:26.142483 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:27:26.142648 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:27:26.142682 (+ 34us) client_negotiation.cc:769] Sending connection context
0929 20:27:26.142765 (+ 83us) client_negotiation.cc:240] Negotiation successful
0929 20:27:26.142917 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":581,"mutex_wait_us":3,"thread_start_us":377,"threads_started":1}
I20260929 20:27:26.142565 3311 tablet_service.cc:1467] Processing CreateTablet for tablet fc1c9c0317ff48939844703bb04b1d2c (DEFAULT_TABLE table=test-table [id=6c29b3be2a3b4659a5c88867099c63be]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:27:26.143569 3392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:27:26.128879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39292 (local address 127.17.53.65:42299)
0929 20:27:26.133482 (+ 4603us) server_negotiation.cc:206] Beginning negotiation
0929 20:27:26.133490 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 20:27:26.135301 (+ 1811us) server_negotiation.cc:407] Connection header received
0929 20:27:26.135498 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:27:26.135507 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:27:26.135823 (+ 316us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:27:26.136025 (+ 202us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:27:26.136756 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:27:26.138289 (+ 1533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:27:26.140838 (+ 2549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:27:26.142256 (+ 1418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:27:26.142408 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:27:26.142965 (+ 557us) server_negotiation.cc:1035] Waiting for connection context
0929 20:27:26.143039 (+ 74us) server_negotiation.cc:299] Negotiation successful
0929 20:27:26.143201 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4420,"thread_start_us":4297,"threads_started":1}
I20260929 20:27:26.144160 3181 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc1c9c0317ff48939844703bb04b1d2c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:27:26.144290 3311 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc1c9c0317ff48939844703bb04b1d2c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:27:26.147643 3048 tablet_service.cc:1467] Processing CreateTablet for tablet fc1c9c0317ff48939844703bb04b1d2c (DEFAULT_TABLE table=test-table [id=6c29b3be2a3b4659a5c88867099c63be]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:27:26.149151 3048 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc1c9c0317ff48939844703bb04b1d2c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:27:26.153155 3114 heartbeater.cc:503] Master 127.17.53.126:43737 was elected leader, sending a full tablet report...
I20260929 20:27:26.156886 3394 tablet_bootstrap.cc:492] T fc1c9c0317ff48939844703bb04b1d2c P 5f877d75d3b345bab8d9a9981ea9690c: Bootstrap starting.
I20260929 20:27:26.158630 3395 tablet_bootstrap.cc:492] T fc1c9c0317ff48939844703bb04b1d2c P 04e06a44c1b64960b4f41240d5fc7f16: Bootstrap starting.
I20260929 20:27:26.161995 3394 tablet_bootstrap.cc:654] T fc1c9c0317ff48939844703bb04b1d2c P 5f877d75d3b345bab8d9a9981ea9690c: Neither blocks nor log segments found. Creating new log.
I20260929 20:27:26.162392 3395 tablet_bootstrap.cc:654] T fc1c9c0317ff48939844703bb04b1d2c P 04e06a44c1b64960b4f41240d5fc7f16: Neither blocks nor log segments found. Creating new log.
I20260929 20:27:26.163287 3396 tablet_bootstrap.cc:492] T fc1c9c0317ff48939844703bb04b1d2c P a939b910230949ed8f99749dd8b8c84f: Bootstrap starting.
I20260929 20:27:26.163552 3395 log.cc:824] T fc1c9c0317ff48939844703bb04b1d2c P 04e06a44c1b64960b4f41240d5fc7f16: Log is configured to *not* fsync() on all Append() calls
I20260929 20:27:26.163513 3394 log.cc:824] T fc1c9c0317ff48939844703bb04b1d2c P 5f877d75d3b345bab8d9a9981ea9690c: Log is configured to *not* fsync() on all Append() calls
I20260929 20:27:26.166496 3395 tablet_bootstrap.cc:492] T fc1c9c0317ff48939844703bb04b1d2c P 04e06a44c1b64960b4f41240d5fc7f16: No bootstrap required, opened a new log
I20260929 20:27:26.166788 3395 ts_tablet_manager.cc:1397] T fc1c9c0317ff48939844703bb04b1d2c P 04e06a44c1b64960b4f41240d5fc7f16: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20260929 20:27:26.167024 3394 tablet_bootstrap.cc:492] T fc1c9c0317ff48939844703bb04b1d2c P 5f877d75d3b345bab8d9a9981ea9690c: No bootstrap required, opened a new log
I20260929 20:27:26.167006 3396 tablet_bootstrap.cc:654] T fc1c9c0317ff48939844703bb04b1d2c P a939b910230949ed8f99749dd8b8c84f: Neither blocks nor log segments found. Creating new log.
I20260929 20:27:26.167292 3394 ts_tablet_manager.cc:1397] T fc1c9c0317ff48939844703bb04b1d2c P 5f877d75d3b345bab8d9a9981ea9690c: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.000s
I20260929 20:27:26.168239 3396 log.cc:824] T fc1c9c0317ff48939844703bb04b1d2c P a939b910230949ed8f99749dd8b8c84f: Log is configured to *not* fsync() on all Append() calls
I20260929 20:27:26.173616 3396 tablet_bootstrap.cc:492] T fc1c9c0317ff48939844703bb04b1d2c P a939b910230949ed8f99749dd8b8c84f: No bootstrap required, opened a new log
I20260929 20:27:26.173954 3396 ts_tablet_manager.cc:1397] T fc1c9c0317ff48939844703bb04b1d2c P a939b910230949ed8f99749dd8b8c84f: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.009s
I20260929 20:27:26.175477 3395 raft_consensus.cc:348] T fc1c9c0317ff48939844703bb04b1d2c P 04e06a44c1b64960b4f41240d5fc7f16 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f877d75d3b345bab8d9a9981ea9690c" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 40629 } } peers { permanent_uuid: "a939b910230949ed8f99749dd8b8c84f" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 42299 } } peers { permanent_uuid: "04e06a44c1b64960b4f41240d5fc7f16" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 37037 } }
I20260929 20:27:26.175863 3395 raft_consensus.cc:374] T fc1c9c0317ff48939844703bb04b1d2c P 04e06a44c1b64960b4f41240d5fc7f16 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:27:26.176018 3395 raft_consensus.cc:729] T fc1c9c0317ff48939844703bb04b1d2c P 04e06a44c1b64960b4f41240d5fc7f16 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04e06a44c1b64960b4f41240d5fc7f16, State: Initialized, Role: FOLLOWER
I20260929 20:27:26.176656 3395 consensus_queue.cc:260] T fc1c9c0317ff48939844703bb04b1d2c P 04e06a44c1b64960b4f41240d5fc7f16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f877d75d3b345bab8d9a9981ea9690c" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 40629 } } peers { permanent_uuid: "a939b910230949ed8f99749dd8b8c84f" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 42299 } } peers { permanent_uuid: "04e06a44c1b64960b4f41240d5fc7f16" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 37037 } }
I20260929 20:27:26.178570 3376 heartbeater.cc:503] Master 127.17.53.126:43737 was elected leader, sending a full tablet report...
I20260929 20:27:26.178316 3394 raft_consensus.cc:348] T fc1c9c0317ff48939844703bb04b1d2c P 5f877d75d3b345bab8d9a9981ea9690c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f877d75d3b345bab8d9a9981ea9690c" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 40629 } } peers { permanent_uuid: "a939b910230949ed8f99749dd8b8c84f" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 42299 } } peers { permanent_uuid: "04e06a44c1b64960b4f41240d5fc7f16" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 37037 } }
I20260929 20:27:26.178745 3394 raft_consensus.cc:374] T fc1c9c0317ff48939844703bb04b1d2c P 5f877d75d3b345bab8d9a9981ea9690c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:27:26.178841 3394 raft_consensus.cc:729] T fc1c9c0317ff48939844703bb04b1d2c P 5f877d75d3b345bab8d9a9981ea9690c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f877d75d3b345bab8d9a9981ea9690c, State: Initialized, Role: FOLLOWER
I20260929 20:27:26.179446 3394 consensus_queue.cc:260] T fc1c9c0317ff48939844703bb04b1d2c P 5f877d75d3b345bab8d9a9981ea9690c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f877d75d3b345bab8d9a9981ea9690c" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 40629 } } peers { permanent_uuid: "a939b910230949ed8f99749dd8b8c84f" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 42299 } } peers { permanent_uuid: "04e06a44c1b64960b4f41240d5fc7f16" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 37037 } }
I20260929 20:27:26.179903 3395 ts_tablet_manager.cc:1428] T fc1c9c0317ff48939844703bb04b1d2c P 04e06a44c1b64960b4f41240d5fc7f16: Time spent starting tablet: real 0.013s user 0.011s sys 0.001s
I20260929 20:27:26.180823 3246 heartbeater.cc:503] Master 127.17.53.126:43737 was elected leader, sending a full tablet report...
I20260929 20:27:26.181471 3394 ts_tablet_manager.cc:1428] T fc1c9c0317ff48939844703bb04b1d2c P 5f877d75d3b345bab8d9a9981ea9690c: Time spent starting tablet: real 0.014s user 0.010s sys 0.004s
I20260929 20:27:26.184870 3396 raft_consensus.cc:348] T fc1c9c0317ff48939844703bb04b1d2c P a939b910230949ed8f99749dd8b8c84f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f877d75d3b345bab8d9a9981ea9690c" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 40629 } } peers { permanent_uuid: "a939b910230949ed8f99749dd8b8c84f" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 42299 } } peers { permanent_uuid: "04e06a44c1b64960b4f41240d5fc7f16" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 37037 } }
I20260929 20:27:26.185341 3396 raft_consensus.cc:374] T fc1c9c0317ff48939844703bb04b1d2c P a939b910230949ed8f99749dd8b8c84f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:27:26.185448 3396 raft_consensus.cc:729] T fc1c9c0317ff48939844703bb04b1d2c P a939b910230949ed8f99749dd8b8c84f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a939b910230949ed8f99749dd8b8c84f, State: Initialized, Role: FOLLOWER
I20260929 20:27:26.186025 3396 consensus_queue.cc:260] T fc1c9c0317ff48939844703bb04b1d2c P a939b910230949ed8f99749dd8b8c84f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f877d75d3b345bab8d9a9981ea9690c" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 40629 } } peers { permanent_uuid: "a939b910230949ed8f99749dd8b8c84f" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 42299 } } peers { permanent_uuid: "04e06a44c1b64960b4f41240d5fc7f16" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 37037 } }
I20260929 20:27:26.187667 3396 ts_tablet_manager.cc:1428] T fc1c9c0317ff48939844703bb04b1d2c P a939b910230949ed8f99749dd8b8c84f: Time spent starting tablet: real 0.014s user 0.009s sys 0.003s
I20260929 20:27:26.193852 3400 raft_consensus.cc:482] T fc1c9c0317ff48939844703bb04b1d2c P 04e06a44c1b64960b4f41240d5fc7f16 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 20:27:26.194118 3400 raft_consensus.cc:504] T fc1c9c0317ff48939844703bb04b1d2c P 04e06a44c1b64960b4f41240d5fc7f16 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f877d75d3b345bab8d9a9981ea9690c" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 40629 } } peers { permanent_uuid: "a939b910230949ed8f99749dd8b8c84f" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 42299 } } peers { permanent_uuid: "04e06a44c1b64960b4f41240d5fc7f16" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 37037 } }
I20260929 20:27:26.195708 3400 leader_election.cc:288] T fc1c9c0317ff48939844703bb04b1d2c P 04e06a44c1b64960b4f41240d5fc7f16 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5f877d75d3b345bab8d9a9981ea9690c (127.17.53.66:40629), a939b910230949ed8f99749dd8b8c84f (127.17.53.65:42299)
I20260929 20:27:26.201654 3378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:27:26.196095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.66:40629 (local address 127.17.53.67:42731)
0929 20:27:26.196285 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0929 20:27:26.196301 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 20:27:26.196397 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:27:26.196807 (+ 410us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:27:26.196813 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:27:26.196840 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:27:26.197077 (+ 237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:27:26.197096 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:27:26.198356 (+ 1260us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:27:26.198362 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:27:26.199786 (+ 1424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:27:26.199799 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:27:26.200853 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:27:26.200859 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:27:26.201033 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:27:26.201090 (+ 57us) client_negotiation.cc:769] Sending connection context
0929 20:27:26.201217 (+ 127us) client_negotiation.cc:240] Negotiation successful
0929 20:27:26.201364 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59}
I20260929 20:27:26.201650 3390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:27:26.196151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.53.67:42731 (local address 127.17.53.66:40629)
0929 20:27:26.196368 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0929 20:27:26.196373 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:27:26.196395 (+ 22us) server_negotiation.cc:407] Connection header received
0929 20:27:26.196562 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:27:26.196568 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:27:26.196663 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:27:26.196775 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:27:26.197254 (+ 479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:27:26.198219 (+ 965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:27:26.199967 (+ 1748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:27:26.200697 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:27:26.200862 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:27:26.200931 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0929 20:27:26.201256 (+ 325us) server_negotiation.cc:299] Negotiation successful
0929 20:27:26.201380 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260929 20:27:26.203032 3201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc1c9c0317ff48939844703bb04b1d2c" candidate_uuid: "04e06a44c1b64960b4f41240d5fc7f16" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f877d75d3b345bab8d9a9981ea9690c" is_pre_election: true
I20260929 20:27:26.203474 3201 raft_consensus.cc:2454] T fc1c9c0317ff48939844703bb04b1d2c P 5f877d75d3b345bab8d9a9981ea9690c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 04e06a44c1b64960b4f41240d5fc7f16 in term 0.
I20260929 20:27:26.204169 3264 leader_election.cc:302] T fc1c9c0317ff48939844703bb04b1d2c P 04e06a44c1b64960b4f41240d5fc7f16 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 04e06a44c1b64960b4f41240d5fc7f16, 5f877d75d3b345bab8d9a9981ea9690c; no voters:
I20260929 20:27:26.204420 3403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:27:26.196537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.65:42299 (local address 127.17.53.67:53009)
0929 20:27:26.197486 (+ 949us) negotiation.cc:107] Waiting for socket to connect
0929 20:27:26.197500 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 20:27:26.197593 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:27:26.197921 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:27:26.197925 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:27:26.197943 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:27:26.198295 (+ 352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:27:26.198305 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:27:26.199405 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:27:26.199411 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:27:26.200769 (+ 1358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:27:26.200816 (+ 47us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:27:26.203808 (+ 2992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:27:26.203820 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:27:26.203983 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:27:26.204006 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 20:27:26.204080 (+ 74us) client_negotiation.cc:240] Negotiation successful
0929 20:27:26.204196 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":848,"thread_start_us":683,"threads_started":1}
I20260929 20:27:26.204507 3392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:27:26.196714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.53.67:53009 (local address 127.17.53.65:42299)
0929 20:27:26.196979 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0929 20:27:26.196986 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:27:26.197585 (+ 599us) server_negotiation.cc:407] Connection header received
0929 20:27:26.197693 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:27:26.197698 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:27:26.197775 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:27:26.197866 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:27:26.198464 (+ 598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:27:26.199240 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:27:26.200994 (+ 1754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:27:26.202133 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:27:26.202290 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:27:26.202373 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0929 20:27:26.204155 (+ 1782us) server_negotiation.cc:299] Negotiation successful
0929 20:27:26.204290 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260929 20:27:26.204643 3400 raft_consensus.cc:2781] T fc1c9c0317ff48939844703bb04b1d2c P 04e06a44c1b64960b4f41240d5fc7f16 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 20:27:26.204782 3400 raft_consensus.cc:482] T fc1c9c0317ff48939844703bb04b1d2c P 04e06a44c1b64960b4f41240d5fc7f16 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 20:27:26.204882 3400 raft_consensus.cc:3037] T fc1c9c0317ff48939844703bb04b1d2c P 04e06a44c1b64960b4f41240d5fc7f16 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:27:26.205401 3068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc1c9c0317ff48939844703bb04b1d2c" candidate_uuid: "04e06a44c1b64960b4f41240d5fc7f16" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a939b910230949ed8f99749dd8b8c84f" is_pre_election: true
I20260929 20:27:26.205797 3068 raft_consensus.cc:2454] T fc1c9c0317ff48939844703bb04b1d2c P a939b910230949ed8f99749dd8b8c84f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 04e06a44c1b64960b4f41240d5fc7f16 in term 0.
I20260929 20:27:26.207406 3400 raft_consensus.cc:504] T fc1c9c0317ff48939844703bb04b1d2c P 04e06a44c1b64960b4f41240d5fc7f16 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f877d75d3b345bab8d9a9981ea9690c" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 40629 } } peers { permanent_uuid: "a939b910230949ed8f99749dd8b8c84f" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 42299 } } peers { permanent_uuid: "04e06a44c1b64960b4f41240d5fc7f16" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 37037 } }
I20260929 20:27:26.207959 3400 leader_election.cc:288] T fc1c9c0317ff48939844703bb04b1d2c P 04e06a44c1b64960b4f41240d5fc7f16 [CANDIDATE]: Term 1 election: Requested vote from peers 5f877d75d3b345bab8d9a9981ea9690c (127.17.53.66:40629), a939b910230949ed8f99749dd8b8c84f (127.17.53.65:42299)
I20260929 20:27:26.208411 3201 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc1c9c0317ff48939844703bb04b1d2c" candidate_uuid: "04e06a44c1b64960b4f41240d5fc7f16" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f877d75d3b345bab8d9a9981ea9690c"
I20260929 20:27:26.208463 3068 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc1c9c0317ff48939844703bb04b1d2c" candidate_uuid: "04e06a44c1b64960b4f41240d5fc7f16" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a939b910230949ed8f99749dd8b8c84f"
I20260929 20:27:26.208614 3201 raft_consensus.cc:3037] T fc1c9c0317ff48939844703bb04b1d2c P 5f877d75d3b345bab8d9a9981ea9690c [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:27:26.208647 3068 raft_consensus.cc:3037] T fc1c9c0317ff48939844703bb04b1d2c P a939b910230949ed8f99749dd8b8c84f [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:27:26.210783 3068 raft_consensus.cc:2454] T fc1c9c0317ff48939844703bb04b1d2c P a939b910230949ed8f99749dd8b8c84f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 04e06a44c1b64960b4f41240d5fc7f16 in term 1.
I20260929 20:27:26.210783 3201 raft_consensus.cc:2454] T fc1c9c0317ff48939844703bb04b1d2c P 5f877d75d3b345bab8d9a9981ea9690c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 04e06a44c1b64960b4f41240d5fc7f16 in term 1.
I20260929 20:27:26.211186 3264 leader_election.cc:302] T fc1c9c0317ff48939844703bb04b1d2c P 04e06a44c1b64960b4f41240d5fc7f16 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 04e06a44c1b64960b4f41240d5fc7f16, 5f877d75d3b345bab8d9a9981ea9690c; no voters:
I20260929 20:27:26.211483 3400 raft_consensus.cc:2781] T fc1c9c0317ff48939844703bb04b1d2c P 04e06a44c1b64960b4f41240d5fc7f16 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:27:26.211989 3400 raft_consensus.cc:686] T fc1c9c0317ff48939844703bb04b1d2c P 04e06a44c1b64960b4f41240d5fc7f16 [term 1 LEADER]: Becoming Leader. State: Replica: 04e06a44c1b64960b4f41240d5fc7f16, State: Running, Role: LEADER
I20260929 20:27:26.212450 3400 consensus_queue.cc:237] T fc1c9c0317ff48939844703bb04b1d2c P 04e06a44c1b64960b4f41240d5fc7f16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f877d75d3b345bab8d9a9981ea9690c" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 40629 } } peers { permanent_uuid: "a939b910230949ed8f99749dd8b8c84f" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 42299 } } peers { permanent_uuid: "04e06a44c1b64960b4f41240d5fc7f16" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 37037 } }
I20260929 20:27:26.217242 2926 catalog_manager.cc:5495] T fc1c9c0317ff48939844703bb04b1d2c P 04e06a44c1b64960b4f41240d5fc7f16 reported cstate change: term changed from 0 to 1, leader changed from <none> to 04e06a44c1b64960b4f41240d5fc7f16 (127.17.53.67). New cstate: current_term: 1 leader_uuid: "04e06a44c1b64960b4f41240d5fc7f16" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f877d75d3b345bab8d9a9981ea9690c" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 40629 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a939b910230949ed8f99749dd8b8c84f" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 42299 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "04e06a44c1b64960b4f41240d5fc7f16" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 37037 } health_report { overall_health: HEALTHY } } }
I20260929 20:27:26.250538 3391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:27:26.244929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55704 (local address 127.17.53.67:37037)
0929 20:27:26.245165 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0929 20:27:26.245170 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:27:26.245212 (+ 42us) server_negotiation.cc:407] Connection header received
0929 20:27:26.245424 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:27:26.245428 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:27:26.245504 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:27:26.245628 (+ 124us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 20:27:26.246105 (+ 477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:27:26.246825 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:27:26.248387 (+ 1562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:27:26.248784 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:27:26.248885 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:27:26.249432 (+ 547us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 20:27:26.249951 (+ 519us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 20:27:26.250068 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0929 20:27:26.250199 (+ 131us) server_negotiation.cc:299] Negotiation successful
0929 20:27:26.250259 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260929 20:27:26.258782 3201 raft_consensus.cc:1261] T fc1c9c0317ff48939844703bb04b1d2c P 5f877d75d3b345bab8d9a9981ea9690c [term 1 FOLLOWER]: Refusing update from remote peer 04e06a44c1b64960b4f41240d5fc7f16: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:27:26.258813 3068 raft_consensus.cc:1261] T fc1c9c0317ff48939844703bb04b1d2c P a939b910230949ed8f99749dd8b8c84f [term 1 FOLLOWER]: Refusing update from remote peer 04e06a44c1b64960b4f41240d5fc7f16: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:27:26.259758 3400 consensus_queue.cc:1035] T fc1c9c0317ff48939844703bb04b1d2c P 04e06a44c1b64960b4f41240d5fc7f16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a939b910230949ed8f99749dd8b8c84f" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 42299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:27:26.260111 3404 consensus_queue.cc:1035] T fc1c9c0317ff48939844703bb04b1d2c P 04e06a44c1b64960b4f41240d5fc7f16 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f877d75d3b345bab8d9a9981ea9690c" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 40629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:27:26.276976 3408 mvcc.cc:205] Tried to move back new op lower bound from 7334763095065894912 to 7334763094890221568. Current Snapshot: MvccSnapshot[applied={T|T < 7334763095065894912}]
I20260929 20:27:26.286762 2926 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:45170:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 20:27:26.287186 2926 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:45170:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 20:27:26.291004 2926 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3a7c2d3af7964f57a509d9f479725278: Sending DeleteTablet for 3 replicas of tablet fc1c9c0317ff48939844703bb04b1d2c
I20260929 20:27:26.292236 3048 tablet_service.cc:1514] Processing DeleteTablet for tablet fc1c9c0317ff48939844703bb04b1d2c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:27:26 UTC) from {username='slave'} at 127.0.0.1:39292
I20260929 20:27:26.292249 3181 tablet_service.cc:1514] Processing DeleteTablet for tablet fc1c9c0317ff48939844703bb04b1d2c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:27:26 UTC) from {username='slave'} at 127.0.0.1:52864
I20260929 20:27:26.292301 3311 tablet_service.cc:1514] Processing DeleteTablet for tablet fc1c9c0317ff48939844703bb04b1d2c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:27:26 UTC) from {username='slave'} at 127.0.0.1:55694
I20260929 20:27:26.295159 17621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu pid 2986
I20260929 20:27:26.313275 3416 tablet_replica.cc:330] T fc1c9c0317ff48939844703bb04b1d2c P a939b910230949ed8f99749dd8b8c84f: stopping tablet replica
I20260929 20:27:26.312989 3417 tablet_replica.cc:330] T fc1c9c0317ff48939844703bb04b1d2c P 04e06a44c1b64960b4f41240d5fc7f16: stopping tablet replica
I20260929 20:27:26.313537 3418 tablet_replica.cc:330] T fc1c9c0317ff48939844703bb04b1d2c P 5f877d75d3b345bab8d9a9981ea9690c: stopping tablet replica
I20260929 20:27:26.313798 3417 raft_consensus.cc:2229] T fc1c9c0317ff48939844703bb04b1d2c P 04e06a44c1b64960b4f41240d5fc7f16 [term 1 LEADER]: Raft consensus shutting down.
I20260929 20:27:26.313951 3416 raft_consensus.cc:2229] T fc1c9c0317ff48939844703bb04b1d2c P a939b910230949ed8f99749dd8b8c84f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 20:27:26.313953 3418 raft_consensus.cc:2229] T fc1c9c0317ff48939844703bb04b1d2c P 5f877d75d3b345bab8d9a9981ea9690c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 20:27:26.314494 3416 raft_consensus.cc:2258] T fc1c9c0317ff48939844703bb04b1d2c P a939b910230949ed8f99749dd8b8c84f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:27:26.314494 3418 raft_consensus.cc:2258] T fc1c9c0317ff48939844703bb04b1d2c P 5f877d75d3b345bab8d9a9981ea9690c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:27:26.314572 3417 raft_consensus.cc:2258] T fc1c9c0317ff48939844703bb04b1d2c P 04e06a44c1b64960b4f41240d5fc7f16 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:27:26.316810 3418 ts_tablet_manager.cc:1905] T fc1c9c0317ff48939844703bb04b1d2c P 5f877d75d3b345bab8d9a9981ea9690c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:27:26.316895 3417 ts_tablet_manager.cc:1905] T fc1c9c0317ff48939844703bb04b1d2c P 04e06a44c1b64960b4f41240d5fc7f16: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:27:26.317037 3392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:27:26.296918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39308 (local address 127.17.53.65:42299)
0929 20:27:26.297148 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0929 20:27:26.297158 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 20:27:26.301003 (+ 3845us) server_negotiation.cc:407] Connection header received
0929 20:27:26.302494 (+ 1491us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:27:26.302501 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:27:26.302624 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:27:26.302705 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:27:26.304324 (+ 1619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:27:26.305561 (+ 1237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:27:26.310167 (+ 4606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:27:26.310906 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:27:26.311026 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:27:26.315275 (+ 4249us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:27:26.315299 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:27:26.315316 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:27:26.316090 (+ 774us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:27:26.316163 (+ 73us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:27:26.316190 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:27:26.316308 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0929 20:27:26.316573 (+ 265us) server_negotiation.cc:299] Negotiation successful
0929 20:27:26.316696 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260929 20:27:26.318266 3109 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:27:26.325666 3418 ts_tablet_manager.cc:1918] T fc1c9c0317ff48939844703bb04b1d2c P 5f877d75d3b345bab8d9a9981ea9690c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 20:27:26.325871 3418 log.cc:1196] T fc1c9c0317ff48939844703bb04b1d2c P 5f877d75d3b345bab8d9a9981ea9690c: Deleting WAL directory at /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790713339161492-17621-0/minicluster-data/ts-1/wal/wals/fc1c9c0317ff48939844703bb04b1d2c
I20260929 20:27:26.325855 3417 ts_tablet_manager.cc:1918] T fc1c9c0317ff48939844703bb04b1d2c P 04e06a44c1b64960b4f41240d5fc7f16: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 20:27:26.326020 3417 log.cc:1196] T fc1c9c0317ff48939844703bb04b1d2c P 04e06a44c1b64960b4f41240d5fc7f16: Deleting WAL directory at /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790713339161492-17621-0/minicluster-data/ts-2/wal/wals/fc1c9c0317ff48939844703bb04b1d2c
I20260929 20:27:26.326474 3418 ts_tablet_manager.cc:1939] T fc1c9c0317ff48939844703bb04b1d2c P 5f877d75d3b345bab8d9a9981ea9690c: Deleting consensus metadata
I20260929 20:27:26.326699 3417 ts_tablet_manager.cc:1939] T fc1c9c0317ff48939844703bb04b1d2c P 04e06a44c1b64960b4f41240d5fc7f16: Deleting consensus metadata
I20260929 20:27:26.328579 2913 catalog_manager.cc:4841] TS 04e06a44c1b64960b4f41240d5fc7f16 (127.17.53.67:37037): tablet fc1c9c0317ff48939844703bb04b1d2c (table test-table [id=6c29b3be2a3b4659a5c88867099c63be]) successfully deleted
I20260929 20:27:26.328606 2911 catalog_manager.cc:4841] TS 5f877d75d3b345bab8d9a9981ea9690c (127.17.53.66:40629): tablet fc1c9c0317ff48939844703bb04b1d2c (table test-table [id=6c29b3be2a3b4659a5c88867099c63be]) successfully deleted
I20260929 20:27:26.319345 3416 ts_tablet_manager.cc:1905] T fc1c9c0317ff48939844703bb04b1d2c P a939b910230949ed8f99749dd8b8c84f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:27:26.417901 17621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu with pid 2986
W20260929 20:27:26.443969 2911 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.17.53.65:42299 (error 108)
I20260929 20:27:26.444242 17621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu pid 3118
I20260929 20:27:26.446189 3393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:27:26.445647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.65:42299 (local address 127.0.0.1:39318)
0929 20:27:26.445854 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0929 20:27:26.446001 (+ 147us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.53.65:42299: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
W20260929 20:27:26.446519 2911 catalog_manager.cc:4584] TS a939b910230949ed8f99749dd8b8c84f (127.17.53.65:42299): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet fc1c9c0317ff48939844703bb04b1d2c: Network error: Client connection negotiation failed: client connection to 127.17.53.65:42299: connect: Connection refused (error 111)
I20260929 20:27:26.451756 3390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:27:26.445190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52874 (local address 127.17.53.66:40629)
0929 20:27:26.445391 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0929 20:27:26.445395 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:27:26.445414 (+ 19us) server_negotiation.cc:407] Connection header received
0929 20:27:26.445648 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:27:26.445654 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:27:26.445785 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:27:26.445931 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:27:26.446508 (+ 577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:27:26.447391 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:27:26.448720 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:27:26.449132 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:27:26.449269 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:27:26.450353 (+ 1084us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:27:26.450362 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:27:26.450376 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:27:26.451003 (+ 627us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:27:26.451052 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:27:26.451071 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:27:26.451204 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0929 20:27:26.451342 (+ 138us) server_negotiation.cc:299] Negotiation successful
0929 20:27:26.451445 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260929 20:27:26.452407 3241 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:27:26.500156 3393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:27:26.499618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.65:42299 (local address 127.0.0.1:39322)
0929 20:27:26.499843 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0929 20:27:26.499953 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.53.65:42299: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260929 20:27:26.501335 3393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:27:26.500930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.65:42299 (local address 127.0.0.1:39326)
0929 20:27:26.501094 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0929 20:27:26.501198 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.53.65:42299: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260929 20:27:26.548555 17621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu with pid 3118
I20260929 20:27:26.562306 3393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:27:26.561653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.65:42299 (local address 127.0.0.1:39332)
0929 20:27:26.561980 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0929 20:27:26.562082 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.53.65:42299: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169}
I20260929 20:27:26.563603 3393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:27:26.563178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.65:42299 (local address 127.0.0.1:39344)
0929 20:27:26.563373 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0929 20:27:26.563448 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.53.65:42299: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102}
I20260929 20:27:26.569808 17621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu pid 3249
I20260929 20:27:26.579020 3391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:27:26.570555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55712 (local address 127.17.53.67:37037)
0929 20:27:26.570737 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0929 20:27:26.570743 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:27:26.570954 (+ 211us) server_negotiation.cc:407] Connection header received
0929 20:27:26.571129 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:27:26.571135 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:27:26.571233 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:27:26.571347 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:27:26.572074 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:27:26.573293 (+ 1219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:27:26.575277 (+ 1984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:27:26.575963 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:27:26.576100 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:27:26.577364 (+ 1264us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:27:26.577378 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:27:26.577396 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:27:26.578157 (+ 761us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:27:26.578216 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:27:26.578239 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:27:26.578356 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0929 20:27:26.578529 (+ 173us) server_negotiation.cc:299] Negotiation successful
0929 20:27:26.578628 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260929 20:27:26.579807 3371 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:27:26.644219 3393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:27:26.643691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.65:42299 (local address 127.0.0.1:39352)
0929 20:27:26.643940 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0929 20:27:26.644049 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.53.65:42299: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20260929 20:27:26.645495 3393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:27:26.645054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.65:42299 (local address 127.0.0.1:39366)
0929 20:27:26.645283 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0929 20:27:26.645377 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.53.65:42299: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105}
I20260929 20:27:26.702050 17621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu with pid 3249
I20260929 20:27:26.721942 17621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu pid 2896
I20260929 20:27:26.723353 2957 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:27:26.834637 17621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu with pid 2896
2026-09-29T20:27:26Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (2803 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790713339161492-17621-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 20:27:26 dist-test-slave-zbnc krb5kdc[3427](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 20:27:26 dist-test-slave-zbnc krb5kdc[3427](info): set up 2 sockets
Sep 29 20:27:26 dist-test-slave-zbnc krb5kdc[3427](info): commencing operation
krb5kdc: starting...
W20260929 20:27:27.960640 17621 mini_kdc.cc:121] Time spent starting KDC: real 1.082s user 0.001s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790713339161492-17621-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790713339161492-17621-0/krb5kdc/test-user.keytab.
Sep 29 20:27:28 dist-test-slave-zbnc krb5kdc[3427](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713648, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T20:27:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T20:27:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.17.53.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.53.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.53.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.126.keytab.
Entry for principal kudu/127.17.53.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.126.keytab.
I20260929 20:27:28.215216 17621 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
/tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790713339161492-17621-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790713339161492-17621-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790713339161492-17621-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790713339161492-17621-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.53.126:39989
--webserver_interface=127.17.53.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.126.keytab
--principal=kudu/127.17.53.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.53.84:34391
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.53.126:39989
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790713339161492-17621-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790713339161492-17621-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790713339161492-17621-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:27:28.547284 3440 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:27:28.547617 3440 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:27:28.547773 3440 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:27:28.555403 3440 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:27:28.555548 3440 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:27:28.555651 3440 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:27:28.555756 3440 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:27:28.555823 3440 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:27:28.565712 3440 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.53.84:34391
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790713339161492-17621-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790713339161492-17621-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.53.126:39989
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.126.keytab
--principal=kudu/127.17.53.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.53.126:39989
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790713339161492-17621-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.53.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790713339161492-17621-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:27:28.567148 3440 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:27:28.568490 3440 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:27:28.581179 3446 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:27:28.581100 3445 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:27:28.581156 3448 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:27:28.581612 3440 server_base.cc:1034] running on GCE node
I20260929 20:27:28.582410 3440 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:27:28.583918 3440 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:27:28.585191 3440 hybrid_clock.cc:648] HybridClock initialized: now 1790713648585140 us; error 34 us; skew 500 ppm
I20260929 20:27:28.585599 3440 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:27:28 dist-test-slave-zbnc krb5kdc[3427](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713648, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.53.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:27:28.588927 3440 init.cc:377] Logged in from keytab as kudu/127.17.53.126@KRBTEST.COM (short username kudu)
I20260929 20:27:28.590492 3440 webserver.cc:460] Webserver started at http://127.17.53.126:44091/ using document root <none> and password file <none>
I20260929 20:27:28.591259 3440 fs_manager.cc:359] Metadata directory not provided
I20260929 20:27:28.591403 3440 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:27:28.591719 3440 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:27:28.594745 3440 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790713339161492-17621-0/minicluster-data/master-0/data/instance:
uuid: "52ae69cb81aa4706bf6f6f1dd771c948"
format_stamp: "Formatted at 2026-09-29 20:27:28 on dist-test-slave-zbnc"
server_key: "4c125602ac64271dc5f39dcd384f9154"
server_key_iv: "ece43bf5a8368404da0bf786deb88b6b"
server_key_version: "encryptionkey@0"
I20260929 20:27:28.595593 3440 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790713339161492-17621-0/minicluster-data/master-0/wal/instance:
uuid: "52ae69cb81aa4706bf6f6f1dd771c948"
format_stamp: "Formatted at 2026-09-29 20:27:28 on dist-test-slave-zbnc"
server_key: "4c125602ac64271dc5f39dcd384f9154"
server_key_iv: "ece43bf5a8368404da0bf786deb88b6b"
server_key_version: "encryptionkey@0"
I20260929 20:27:28.600771 3440 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 20:27:28.604115 3455 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:27:28.605588 3440 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 20:27:28.605786 3440 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790713339161492-17621-0/minicluster-data/master-0/data,/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790713339161492-17621-0/minicluster-data/master-0/wal
uuid: "52ae69cb81aa4706bf6f6f1dd771c948"
format_stamp: "Formatted at 2026-09-29 20:27:28 on dist-test-slave-zbnc"
server_key: "4c125602ac64271dc5f39dcd384f9154"
server_key_iv: "ece43bf5a8368404da0bf786deb88b6b"
server_key_version: "encryptionkey@0"
I20260929 20:27:28.605983 3440 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790713339161492-17621-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790713339161492-17621-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790713339161492-17621-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:27:28.631451 3440 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:27:28.635422 3440 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:27:28.635732 3440 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:27:28.654278 3440 rpc_server.cc:307] RPC server started. Bound to: 127.17.53.126:39989
I20260929 20:27:28.654299 3506 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.53.126:39989 every 8 connection(s)
I20260929 20:27:28.655717 3440 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790713339161492-17621-0/minicluster-data/master-0/data/info.pb
I20260929 20:27:28.659174 17621 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu as pid 3440
I20260929 20:27:28.659360 17621 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790713339161492-17621-0/minicluster-data/master-0/wal/instance
I20260929 20:27:28.659703 3507 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:27:28.659965 17621 external_mini_cluster.cc:1428] Setting key 66387c28864e0d37efd9b7e71265bb7e
I20260929 20:27:28.674664 3507 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 52ae69cb81aa4706bf6f6f1dd771c948: Bootstrap starting.
Sep 29 20:27:28 dist-test-slave-zbnc krb5kdc[3427](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713648, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.53.126@KRBTEST.COM
I20260929 20:27:28.678755 3507 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 52ae69cb81aa4706bf6f6f1dd771c948: Neither blocks nor log segments found. Creating new log.
I20260929 20:27:28.680120 3507 log.cc:824] T 00000000000000000000000000000000 P 52ae69cb81aa4706bf6f6f1dd771c948: Log is configured to *not* fsync() on all Append() calls
I20260929 20:27:28.683833 3507 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 52ae69cb81aa4706bf6f6f1dd771c948: No bootstrap required, opened a new log
I20260929 20:27:28.685564 3510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:27:28.661865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52108 (local address 127.17.53.126:39989)
0929 20:27:28.662686 (+ 821us) server_negotiation.cc:206] Beginning negotiation
0929 20:27:28.662698 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 20:27:28.662744 (+ 46us) server_negotiation.cc:407] Connection header received
0929 20:27:28.664614 (+ 1870us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:27:28.664636 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:27:28.665056 (+ 420us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:27:28.665392 (+ 336us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:27:28.667169 (+ 1777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:27:28.668501 (+ 1332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:27:28.670403 (+ 1902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:27:28.671214 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:27:28.671376 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:27:28.676361 (+ 4985us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:27:28.676376 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:27:28.676402 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:27:28.676460 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:27:28.680455 (+ 3995us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:27:28.681671 (+ 1216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:27:28.681681 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:27:28.681688 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:27:28.681893 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:27:28.682329 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:27:28.682333 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:27:28.682336 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:27:28.683047 (+ 711us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:27:28.683348 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0929 20:27:28.683767 (+ 419us) server_negotiation.cc:299] Negotiation successful
0929 20:27:28.684223 (+ 456us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":610,"thread_start_us":398,"threads_started":1}
I20260929 20:27:28.692253 3507 raft_consensus.cc:348] T 00000000000000000000000000000000 P 52ae69cb81aa4706bf6f6f1dd771c948 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52ae69cb81aa4706bf6f6f1dd771c948" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 39989 } }
I20260929 20:27:28.692600 3507 raft_consensus.cc:374] T 00000000000000000000000000000000 P 52ae69cb81aa4706bf6f6f1dd771c948 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:27:28.692703 3507 raft_consensus.cc:729] T 00000000000000000000000000000000 P 52ae69cb81aa4706bf6f6f1dd771c948 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52ae69cb81aa4706bf6f6f1dd771c948, State: Initialized, Role: FOLLOWER
I20260929 20:27:28.693204 3507 consensus_queue.cc:260] T 00000000000000000000000000000000 P 52ae69cb81aa4706bf6f6f1dd771c948 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52ae69cb81aa4706bf6f6f1dd771c948" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 39989 } }
I20260929 20:27:28.693406 3507 raft_consensus.cc:388] T 00000000000000000000000000000000 P 52ae69cb81aa4706bf6f6f1dd771c948 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 20:27:28.693499 3507 raft_consensus.cc:482] T 00000000000000000000000000000000 P 52ae69cb81aa4706bf6f6f1dd771c948 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 20:27:28.693621 3507 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 52ae69cb81aa4706bf6f6f1dd771c948 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:27:28.696084 3507 raft_consensus.cc:504] T 00000000000000000000000000000000 P 52ae69cb81aa4706bf6f6f1dd771c948 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52ae69cb81aa4706bf6f6f1dd771c948" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 39989 } }
I20260929 20:27:28.696521 3507 leader_election.cc:302] T 00000000000000000000000000000000 P 52ae69cb81aa4706bf6f6f1dd771c948 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 52ae69cb81aa4706bf6f6f1dd771c948; no voters:
I20260929 20:27:28.697067 3507 leader_election.cc:288] T 00000000000000000000000000000000 P 52ae69cb81aa4706bf6f6f1dd771c948 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 20:27:28.697219 3512 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 52ae69cb81aa4706bf6f6f1dd771c948 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:27:28.698663 3512 raft_consensus.cc:686] T 00000000000000000000000000000000 P 52ae69cb81aa4706bf6f6f1dd771c948 [term 1 LEADER]: Becoming Leader. State: Replica: 52ae69cb81aa4706bf6f6f1dd771c948, State: Running, Role: LEADER
I20260929 20:27:28.699301 3507 sys_catalog.cc:564] T 00000000000000000000000000000000 P 52ae69cb81aa4706bf6f6f1dd771c948 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:27:28.699213 3512 consensus_queue.cc:237] T 00000000000000000000000000000000 P 52ae69cb81aa4706bf6f6f1dd771c948 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52ae69cb81aa4706bf6f6f1dd771c948" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 39989 } }
I20260929 20:27:28.702011 3514 sys_catalog.cc:455] T 00000000000000000000000000000000 P 52ae69cb81aa4706bf6f6f1dd771c948 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 52ae69cb81aa4706bf6f6f1dd771c948. Latest consensus state: current_term: 1 leader_uuid: "52ae69cb81aa4706bf6f6f1dd771c948" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52ae69cb81aa4706bf6f6f1dd771c948" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 39989 } } }
I20260929 20:27:28.702235 3513 sys_catalog.cc:455] T 00000000000000000000000000000000 P 52ae69cb81aa4706bf6f6f1dd771c948 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "52ae69cb81aa4706bf6f6f1dd771c948" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "52ae69cb81aa4706bf6f6f1dd771c948" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 39989 } } }
I20260929 20:27:28.702518 3514 sys_catalog.cc:458] T 00000000000000000000000000000000 P 52ae69cb81aa4706bf6f6f1dd771c948 [sys.catalog]: This master's current role is: LEADER
I20260929 20:27:28.702555 3513 sys_catalog.cc:458] T 00000000000000000000000000000000 P 52ae69cb81aa4706bf6f6f1dd771c948 [sys.catalog]: This master's current role is: LEADER
I20260929 20:27:28.704859 3518 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:27:28.709586 3518 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:27:28.717746 3518 catalog_manager.cc:1347] Generated new cluster ID: 26582dd360d54fec80dc73da372a8ca8
I20260929 20:27:28.717859 3518 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:27:28.766211 3518 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 20:27:28.767623 3518 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:27:28.773192 3518 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 52ae69cb81aa4706bf6f6f1dd771c948: Generated new TSK 0
I20260929 20:27:28.774258 3518 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.17.53.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.53.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.53.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.65.keytab.
Entry for principal kudu/127.17.53.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.65.keytab.
I20260929 20:27:28.842382 17621 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
/tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790713339161492-17621-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790713339161492-17621-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790713339161492-17621-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790713339161492-17621-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.53.65:0
--local_ip_for_outbound_sockets=127.17.53.65
--webserver_interface=127.17.53.65
--webserver_port=0
--tserver_master_addrs=127.17.53.126:39989
--keytab_file=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.65.keytab
--principal=kudu/127.17.53.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.53.84:34391
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790713339161492-17621-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790713339161492-17621-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790713339161492-17621-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:27:29.184031 3533 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:27:29.184373 3533 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:27:29.184540 3533 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:27:29.191962 3533 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:27:29.192117 3533 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:27:29.192287 3533 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.53.65
I20260929 20:27:29.202556 3533 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.53.84:34391
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790713339161492-17621-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790713339161492-17621-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.65.keytab
--principal=kudu/127.17.53.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.53.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790713339161492-17621-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.17.53.65
--webserver_port=0
--tserver_master_addrs=127.17.53.126:39989
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.53.65
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790713339161492-17621-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:27:29.204160 3533 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:27:29.205603 3533 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:27:29.219512 3539 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:27:29.219549 3541 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:27:29.219462 3538 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:27:29.220759 3533 server_base.cc:1034] running on GCE node
I20260929 20:27:29.221414 3533 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:27:29.222414 3533 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:27:29.223652 3533 hybrid_clock.cc:648] HybridClock initialized: now 1790713649223604 us; error 44 us; skew 500 ppm
I20260929 20:27:29.223978 3533 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:27:29 dist-test-slave-zbnc krb5kdc[3427](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713649, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.53.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:27:29.227193 3533 init.cc:377] Logged in from keytab as kudu/127.17.53.65@KRBTEST.COM (short username kudu)
I20260929 20:27:29.228669 3533 webserver.cc:460] Webserver started at http://127.17.53.65:36305/ using document root <none> and password file <none>
I20260929 20:27:29.229377 3533 fs_manager.cc:359] Metadata directory not provided
I20260929 20:27:29.229530 3533 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:27:29.229854 3533 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:27:29.232959 3533 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790713339161492-17621-0/minicluster-data/ts-0/data/instance:
uuid: "33b49bd6e11e4391b4928e0861f91f59"
format_stamp: "Formatted at 2026-09-29 20:27:29 on dist-test-slave-zbnc"
server_key: "f9be233ecff9e5b818d97b51d221551b"
server_key_iv: "5b4154afdf9dc0a215cfdd3e371c118a"
server_key_version: "encryptionkey@0"
I20260929 20:27:29.233749 3533 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790713339161492-17621-0/minicluster-data/ts-0/wal/instance:
uuid: "33b49bd6e11e4391b4928e0861f91f59"
format_stamp: "Formatted at 2026-09-29 20:27:29 on dist-test-slave-zbnc"
server_key: "f9be233ecff9e5b818d97b51d221551b"
server_key_iv: "5b4154afdf9dc0a215cfdd3e371c118a"
server_key_version: "encryptionkey@0"
I20260929 20:27:29.239140 3533 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260929 20:27:29.242599 3548 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:27:29.243978 3533 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 20:27:29.244174 3533 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790713339161492-17621-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790713339161492-17621-0/minicluster-data/ts-0/wal
uuid: "33b49bd6e11e4391b4928e0861f91f59"
format_stamp: "Formatted at 2026-09-29 20:27:29 on dist-test-slave-zbnc"
server_key: "f9be233ecff9e5b818d97b51d221551b"
server_key_iv: "5b4154afdf9dc0a215cfdd3e371c118a"
server_key_version: "encryptionkey@0"
I20260929 20:27:29.244378 3533 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790713339161492-17621-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790713339161492-17621-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790713339161492-17621-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:27:29.260397 3533 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:27:29.264477 3533 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:27:29.264792 3533 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:27:29.265738 3533 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:27:29.267117 3533 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:27:29.267217 3533 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:27:29.267377 3533 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:27:29.267452 3533 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:27:29.301203 3533 rpc_server.cc:307] RPC server started. Bound to: 127.17.53.65:36159
I20260929 20:27:29.301250 3661 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.53.65:36159 every 8 connection(s)
I20260929 20:27:29.302533 3533 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790713339161492-17621-0/minicluster-data/ts-0/data/info.pb
I20260929 20:27:29.309888 17621 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu as pid 3533
I20260929 20:27:29.310140 17621 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790713339161492-17621-0/minicluster-data/ts-0/wal/instance
I20260929 20:27:29.310737 17621 external_mini_cluster.cc:1428] Setting key d3940914e5d3cf9232f3517bf80b7f31
Sep 29 20:27:29 dist-test-slave-zbnc krb5kdc[3427](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713649, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.53.65@KRBTEST.COM for kudu/127.17.53.126@KRBTEST.COM
I20260929 20:27:29.325191 3664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:27:29.305410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.53.65:51501 (local address 127.17.53.126:39989)
0929 20:27:29.305901 (+ 491us) server_negotiation.cc:206] Beginning negotiation
0929 20:27:29.305907 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:27:29.306903 (+ 996us) server_negotiation.cc:407] Connection header received
0929 20:27:29.308752 (+ 1849us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:27:29.308756 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:27:29.308825 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:27:29.308922 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:27:29.310653 (+ 1731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:27:29.311698 (+ 1045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:27:29.313429 (+ 1731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:27:29.314163 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:27:29.314309 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:27:29.318065 (+ 3756us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:27:29.318073 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:27:29.318081 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:27:29.318125 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:27:29.321793 (+ 3668us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:27:29.322790 (+ 997us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:27:29.322796 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:27:29.322799 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:27:29.322999 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:27:29.323488 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:27:29.323492 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:27:29.323494 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:27:29.323760 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:27:29.323886 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0929 20:27:29.324782 (+ 896us) server_negotiation.cc:299] Negotiation successful
0929 20:27:29.324946 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":348,"thread_start_us":216,"threads_started":1}
I20260929 20:27:29.326750 3665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:27:29.306004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:39989 (local address 127.17.53.65:51501)
0929 20:27:29.306743 (+ 739us) negotiation.cc:107] Waiting for socket to connect
0929 20:27:29.306785 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0929 20:27:29.308502 (+ 1717us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:27:29.309124 (+ 622us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:27:29.309132 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:27:29.309740 (+ 608us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:27:29.310467 (+ 727us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:27:29.310493 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:27:29.311882 (+ 1389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:27:29.311886 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:27:29.313284 (+ 1398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:27:29.313296 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:27:29.314392 (+ 1096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:27:29.314397 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:27:29.314591 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:27:29.315285 (+ 694us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:27:29.315307 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:27:29.317830 (+ 2523us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:27:29.321977 (+ 4147us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:27:29.321987 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:27:29.322008 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:27:29.322643 (+ 635us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:27:29.323115 (+ 472us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:27:29.323123 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:27:29.323129 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:27:29.323371 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:27:29.323907 (+ 536us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:27:29.323918 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:27:29.324374 (+ 456us) client_negotiation.cc:769] Sending connection context
0929 20:27:29.324676 (+ 302us) client_negotiation.cc:240] Negotiation successful
0929 20:27:29.325269 (+ 593us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":506,"thread_start_us":309,"threads_started":1}
I20260929 20:27:29.328543 3662 heartbeater.cc:347] Connected to a master server at 127.17.53.126:39989
I20260929 20:27:29.328852 3662 heartbeater.cc:464] Registering TS with master...
I20260929 20:27:29.329581 3662 heartbeater.cc:511] Master 127.17.53.126:39989 requested a full tablet report, sending...
I20260929 20:27:29.331885 3472 ts_manager.cc:194] Registered new tserver with Master: 33b49bd6e11e4391b4928e0861f91f59 (127.17.53.65:36159)
I20260929 20:27:29.334136 3472 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.53.65@KRBTEST.COM'} at 127.17.53.65:51501
WARNING: no policy specified for kudu/127.17.53.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.53.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.53.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.66.keytab.
Entry for principal kudu/127.17.53.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.66.keytab.
I20260929 20:27:29.370191 17621 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
/tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790713339161492-17621-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790713339161492-17621-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790713339161492-17621-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790713339161492-17621-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.53.66:0
--local_ip_for_outbound_sockets=127.17.53.66
--webserver_interface=127.17.53.66
--webserver_port=0
--tserver_master_addrs=127.17.53.126:39989
--keytab_file=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.66.keytab
--principal=kudu/127.17.53.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.53.84:34391
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790713339161492-17621-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790713339161492-17621-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790713339161492-17621-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:27:29.703383 3669 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:27:29.703658 3669 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:27:29.703835 3669 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:27:29.711292 3669 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:27:29.711423 3669 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:27:29.711599 3669 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.53.66
I20260929 20:27:29.721382 3669 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.53.84:34391
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790713339161492-17621-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790713339161492-17621-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.66.keytab
--principal=kudu/127.17.53.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.53.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790713339161492-17621-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.17.53.66
--webserver_port=0
--tserver_master_addrs=127.17.53.126:39989
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.53.66
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790713339161492-17621-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:27:29.722663 3669 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:27:29.724093 3669 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:27:29.737406 3675 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:27:29.737428 3674 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:27:29.738297 3677 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:27:29.738276 3669 server_base.cc:1034] running on GCE node
I20260929 20:27:29.739076 3669 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:27:29.740103 3669 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:27:29.741336 3669 hybrid_clock.cc:648] HybridClock initialized: now 1790713649741276 us; error 46 us; skew 500 ppm
I20260929 20:27:29.741699 3669 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:27:29 dist-test-slave-zbnc krb5kdc[3427](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713649, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.53.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:27:29.749360 3669 init.cc:377] Logged in from keytab as kudu/127.17.53.66@KRBTEST.COM (short username kudu)
I20260929 20:27:29.751108 3669 webserver.cc:460] Webserver started at http://127.17.53.66:43791/ using document root <none> and password file <none>
I20260929 20:27:29.751868 3669 fs_manager.cc:359] Metadata directory not provided
I20260929 20:27:29.752009 3669 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:27:29.752355 3669 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:27:29.755618 3669 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790713339161492-17621-0/minicluster-data/ts-1/data/instance:
uuid: "d8c6f447caae49bea30f52d56902fe5b"
format_stamp: "Formatted at 2026-09-29 20:27:29 on dist-test-slave-zbnc"
server_key: "0ac9bb9fcd3ee39a5768c09600935d34"
server_key_iv: "315a77c9848a09d05eeca5277a2f6a92"
server_key_version: "encryptionkey@0"
I20260929 20:27:29.756402 3669 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790713339161492-17621-0/minicluster-data/ts-1/wal/instance:
uuid: "d8c6f447caae49bea30f52d56902fe5b"
format_stamp: "Formatted at 2026-09-29 20:27:29 on dist-test-slave-zbnc"
server_key: "0ac9bb9fcd3ee39a5768c09600935d34"
server_key_iv: "315a77c9848a09d05eeca5277a2f6a92"
server_key_version: "encryptionkey@0"
I20260929 20:27:29.761649 3669 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260929 20:27:29.765215 3684 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:27:29.766510 3669 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 20:27:29.766669 3669 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790713339161492-17621-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790713339161492-17621-0/minicluster-data/ts-1/wal
uuid: "d8c6f447caae49bea30f52d56902fe5b"
format_stamp: "Formatted at 2026-09-29 20:27:29 on dist-test-slave-zbnc"
server_key: "0ac9bb9fcd3ee39a5768c09600935d34"
server_key_iv: "315a77c9848a09d05eeca5277a2f6a92"
server_key_version: "encryptionkey@0"
I20260929 20:27:29.766892 3669 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790713339161492-17621-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790713339161492-17621-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790713339161492-17621-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:27:29.785749 3669 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:27:29.790148 3669 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:27:29.790446 3669 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:27:29.791491 3669 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:27:29.792850 3669 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:27:29.792922 3669 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:27:29.793090 3669 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:27:29.793141 3669 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:27:29.826507 3669 rpc_server.cc:307] RPC server started. Bound to: 127.17.53.66:38161
I20260929 20:27:29.826539 3797 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.53.66:38161 every 8 connection(s)
I20260929 20:27:29.827857 3669 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790713339161492-17621-0/minicluster-data/ts-1/data/info.pb
I20260929 20:27:29.835770 17621 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu as pid 3669
I20260929 20:27:29.835960 17621 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790713339161492-17621-0/minicluster-data/ts-1/wal/instance
I20260929 20:27:29.836417 17621 external_mini_cluster.cc:1428] Setting key 20e391b5e714c9b07d42eabc2ab9771e
Sep 29 20:27:29 dist-test-slave-zbnc krb5kdc[3427](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713649, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.53.66@KRBTEST.COM for kudu/127.17.53.126@KRBTEST.COM
I20260929 20:27:29.848974 3800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:27:29.830517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.53.66:55737 (local address 127.17.53.126:39989)
0929 20:27:29.831026 (+ 509us) server_negotiation.cc:206] Beginning negotiation
0929 20:27:29.831031 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:27:29.831949 (+ 918us) server_negotiation.cc:407] Connection header received
0929 20:27:29.833783 (+ 1834us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:27:29.833789 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:27:29.833882 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:27:29.834004 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:27:29.835644 (+ 1640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:27:29.836490 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:27:29.838454 (+ 1964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:27:29.838935 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:27:29.839059 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:27:29.842830 (+ 3771us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:27:29.842835 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:27:29.842838 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:27:29.842927 (+ 89us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:27:29.845608 (+ 2681us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:27:29.846457 (+ 849us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:27:29.846464 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:27:29.846470 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:27:29.846602 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:27:29.847113 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:27:29.847118 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:27:29.847121 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:27:29.847391 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:27:29.847543 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0929 20:27:29.848345 (+ 802us) server_negotiation.cc:299] Negotiation successful
0929 20:27:29.848566 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":402,"thread_start_us":246,"threads_started":1}
I20260929 20:27:29.849992 3801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:27:29.831090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:39989 (local address 127.17.53.66:55737)
0929 20:27:29.831754 (+ 664us) negotiation.cc:107] Waiting for socket to connect
0929 20:27:29.831800 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0929 20:27:29.833509 (+ 1709us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:27:29.834239 (+ 730us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:27:29.834247 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:27:29.834765 (+ 518us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:27:29.835454 (+ 689us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:27:29.835482 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:27:29.836654 (+ 1172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:27:29.836659 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:27:29.838309 (+ 1650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:27:29.838321 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:27:29.839163 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:27:29.839170 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:27:29.839402 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:27:29.840155 (+ 753us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:27:29.840182 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:27:29.842601 (+ 2419us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:27:29.845848 (+ 3247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:27:29.845857 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:27:29.845874 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:27:29.846309 (+ 435us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:27:29.846715 (+ 406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:27:29.846720 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:27:29.846723 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:27:29.847002 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:27:29.847539 (+ 537us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:27:29.847547 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:27:29.847891 (+ 344us) client_negotiation.cc:769] Sending connection context
0929 20:27:29.848256 (+ 365us) client_negotiation.cc:240] Negotiation successful
0929 20:27:29.848823 (+ 567us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":444,"thread_start_us":250,"threads_started":1}
I20260929 20:27:29.851523 3798 heartbeater.cc:347] Connected to a master server at 127.17.53.126:39989
I20260929 20:27:29.851835 3798 heartbeater.cc:464] Registering TS with master...
I20260929 20:27:29.852586 3798 heartbeater.cc:511] Master 127.17.53.126:39989 requested a full tablet report, sending...
I20260929 20:27:29.854171 3472 ts_manager.cc:194] Registered new tserver with Master: d8c6f447caae49bea30f52d56902fe5b (127.17.53.66:38161)
I20260929 20:27:29.855475 3472 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.53.66@KRBTEST.COM'} at 127.17.53.66:55737
WARNING: no policy specified for kudu/127.17.53.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.53.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.53.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.67.keytab.
Entry for principal kudu/127.17.53.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.67.keytab.
I20260929 20:27:29.893894 17621 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
/tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790713339161492-17621-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790713339161492-17621-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790713339161492-17621-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790713339161492-17621-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.53.67:0
--local_ip_for_outbound_sockets=127.17.53.67
--webserver_interface=127.17.53.67
--webserver_port=0
--tserver_master_addrs=127.17.53.126:39989
--keytab_file=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.67.keytab
--principal=kudu/127.17.53.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.53.84:34391
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790713339161492-17621-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790713339161492-17621-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790713339161492-17621-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:27:30.226570 3804 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:27:30.227036 3804 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:27:30.227234 3804 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:27:30.234673 3804 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:27:30.234786 3804 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:27:30.235036 3804 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.53.67
I20260929 20:27:30.245010 3804 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.53.84:34391
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790713339161492-17621-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790713339161492-17621-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.67.keytab
--principal=kudu/127.17.53.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.53.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790713339161492-17621-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.17.53.67
--webserver_port=0
--tserver_master_addrs=127.17.53.126:39989
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.53.67
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790713339161492-17621-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:27:30.246371 3804 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:27:30.247819 3804 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:27:30.260955 3810 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:27:30.261023 3809 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:27:30.261054 3812 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:27:30.262321 3804 server_base.cc:1034] running on GCE node
I20260929 20:27:30.263023 3804 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:27:30.263973 3804 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:27:30.265196 3804 hybrid_clock.cc:648] HybridClock initialized: now 1790713650265207 us; error 108 us; skew 500 ppm
I20260929 20:27:30.265524 3804 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:27:30 dist-test-slave-zbnc krb5kdc[3427](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713650, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.53.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:27:30.268904 3804 init.cc:377] Logged in from keytab as kudu/127.17.53.67@KRBTEST.COM (short username kudu)
I20260929 20:27:30.270428 3804 webserver.cc:460] Webserver started at http://127.17.53.67:44187/ using document root <none> and password file <none>
I20260929 20:27:30.271170 3804 fs_manager.cc:359] Metadata directory not provided
I20260929 20:27:30.271320 3804 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:27:30.271595 3804 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:27:30.274428 3804 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790713339161492-17621-0/minicluster-data/ts-2/data/instance:
uuid: "1da255aea18549f790edb7521f209ec8"
format_stamp: "Formatted at 2026-09-29 20:27:30 on dist-test-slave-zbnc"
server_key: "112518c48721e0b52c6a64832c65d724"
server_key_iv: "211058146f7b4859ccf9883741939bd9"
server_key_version: "encryptionkey@0"
I20260929 20:27:30.275187 3804 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790713339161492-17621-0/minicluster-data/ts-2/wal/instance:
uuid: "1da255aea18549f790edb7521f209ec8"
format_stamp: "Formatted at 2026-09-29 20:27:30 on dist-test-slave-zbnc"
server_key: "112518c48721e0b52c6a64832c65d724"
server_key_iv: "211058146f7b4859ccf9883741939bd9"
server_key_version: "encryptionkey@0"
I20260929 20:27:30.280251 3804 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260929 20:27:30.284183 3819 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:27:30.285528 3804 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 20:27:30.285699 3804 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790713339161492-17621-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790713339161492-17621-0/minicluster-data/ts-2/wal
uuid: "1da255aea18549f790edb7521f209ec8"
format_stamp: "Formatted at 2026-09-29 20:27:30 on dist-test-slave-zbnc"
server_key: "112518c48721e0b52c6a64832c65d724"
server_key_iv: "211058146f7b4859ccf9883741939bd9"
server_key_version: "encryptionkey@0"
I20260929 20:27:30.285879 3804 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790713339161492-17621-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790713339161492-17621-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790713339161492-17621-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:27:30.298280 3804 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:27:30.302814 3804 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:27:30.303184 3804 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:27:30.304160 3804 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:27:30.305701 3804 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:27:30.305806 3804 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:27:30.305949 3804 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:27:30.306031 3804 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:27:30.337324 3662 heartbeater.cc:503] Master 127.17.53.126:39989 was elected leader, sending a full tablet report...
I20260929 20:27:30.341133 3804 rpc_server.cc:307] RPC server started. Bound to: 127.17.53.67:36899
I20260929 20:27:30.341178 3931 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.53.67:36899 every 8 connection(s)
I20260929 20:27:30.342432 3804 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790713339161492-17621-0/minicluster-data/ts-2/data/info.pb
I20260929 20:27:30.349011 17621 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu as pid 3804
I20260929 20:27:30.349223 17621 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790713339161492-17621-0/minicluster-data/ts-2/wal/instance
I20260929 20:27:30.349714 17621 external_mini_cluster.cc:1428] Setting key 3b0f32eead0bca9f06404ea9064ffd0e
Sep 29 20:27:30 dist-test-slave-zbnc krb5kdc[3427](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713650, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.53.67@KRBTEST.COM for kudu/127.17.53.126@KRBTEST.COM
I20260929 20:27:30.362072 3800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:27:30.344947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.53.67:49711 (local address 127.17.53.126:39989)
0929 20:27:30.345130 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0929 20:27:30.345135 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:27:30.346411 (+ 1276us) server_negotiation.cc:407] Connection header received
0929 20:27:30.348355 (+ 1944us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:27:30.348360 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:27:30.348445 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:27:30.348529 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:27:30.350321 (+ 1792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:27:30.351354 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:27:30.353191 (+ 1837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:27:30.353583 (+ 392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:27:30.353681 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:27:30.357018 (+ 3337us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:27:30.357023 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:27:30.357026 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:27:30.357056 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:27:30.359549 (+ 2493us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:27:30.360207 (+ 658us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:27:30.360211 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:27:30.360213 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:27:30.360320 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:27:30.360676 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:27:30.360679 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:27:30.360680 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:27:30.360933 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:27:30.361051 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0929 20:27:30.361645 (+ 594us) server_negotiation.cc:299] Negotiation successful
0929 20:27:30.361805 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260929 20:27:30.363235 3934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:27:30.345478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:39989 (local address 127.17.53.67:49711)
0929 20:27:30.346246 (+ 768us) negotiation.cc:107] Waiting for socket to connect
0929 20:27:30.346294 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0929 20:27:30.348098 (+ 1804us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:27:30.348835 (+ 737us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:27:30.348843 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:27:30.349477 (+ 634us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:27:30.350138 (+ 661us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:27:30.350163 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:27:30.351520 (+ 1357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:27:30.351526 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:27:30.353067 (+ 1541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:27:30.353080 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:27:30.353762 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:27:30.353766 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:27:30.353955 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:27:30.354649 (+ 694us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:27:30.354671 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:27:30.356836 (+ 2165us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:27:30.359678 (+ 2842us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:27:30.359684 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:27:30.359697 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:27:30.360111 (+ 414us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:27:30.360406 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:27:30.360409 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:27:30.360412 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:27:30.360590 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:27:30.361041 (+ 451us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:27:30.361046 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:27:30.361346 (+ 300us) client_negotiation.cc:769] Sending connection context
0929 20:27:30.361567 (+ 221us) client_negotiation.cc:240] Negotiation successful
0929 20:27:30.361942 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":540,"thread_start_us":345,"threads_started":1}
I20260929 20:27:30.364712 3932 heartbeater.cc:347] Connected to a master server at 127.17.53.126:39989
I20260929 20:27:30.365041 3932 heartbeater.cc:464] Registering TS with master...
I20260929 20:27:30.365764 3932 heartbeater.cc:511] Master 127.17.53.126:39989 requested a full tablet report, sending...
I20260929 20:27:30.367156 3472 ts_manager.cc:194] Registered new tserver with Master: 1da255aea18549f790edb7521f209ec8 (127.17.53.67:36899)
I20260929 20:27:30.367944 3472 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.53.67@KRBTEST.COM'} at 127.17.53.67:49711
I20260929 20:27:30.377038 17621 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 20:27:30.393920 3800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:27:30.381061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40756 (local address 127.17.53.126:39989)
0929 20:27:30.381227 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0929 20:27:30.381231 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:27:30.381732 (+ 501us) server_negotiation.cc:407] Connection header received
0929 20:27:30.381865 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:27:30.381868 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:27:30.381937 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:27:30.382016 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:27:30.383403 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:27:30.384129 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:27:30.385401 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:27:30.385812 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:27:30.385908 (+ 96us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:27:30.388598 (+ 2690us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:27:30.388603 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:27:30.388606 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:27:30.388634 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:27:30.391496 (+ 2862us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:27:30.392232 (+ 736us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:27:30.392236 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:27:30.392238 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:27:30.392350 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:27:30.392772 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:27:30.392776 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:27:30.392777 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:27:30.393057 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:27:30.393259 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0929 20:27:30.393546 (+ 287us) server_negotiation.cc:299] Negotiation successful
0929 20:27:30.393670 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 20:27:30.397889 3472 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40756:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 20:27:30.405843 3472 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 20:27:30.430121 3944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:27:30.418312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.65:36159 (local address 127.0.0.1:52162)
0929 20:27:30.420010 (+ 1698us) negotiation.cc:107] Waiting for socket to connect
0929 20:27:30.420068 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0929 20:27:30.420276 (+ 208us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:27:30.421870 (+ 1594us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:27:30.421878 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:27:30.421921 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:27:30.422437 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:27:30.422453 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:27:30.424497 (+ 2044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:27:30.424506 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:27:30.426919 (+ 2413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:27:30.426934 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:27:30.428803 (+ 1869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:27:30.428810 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:27:30.429047 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:27:30.429119 (+ 72us) client_negotiation.cc:769] Sending connection context
0929 20:27:30.429250 (+ 131us) client_negotiation.cc:240] Negotiation successful
0929 20:27:30.429522 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1396,"mutex_wait_us":89,"thread_start_us":1237,"threads_started":1}
I20260929 20:27:30.430289 3947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:27:30.419172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48412 (local address 127.17.53.67:36899)
0929 20:27:30.420725 (+ 1553us) server_negotiation.cc:206] Beginning negotiation
0929 20:27:30.420734 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 20:27:30.421172 (+ 438us) server_negotiation.cc:407] Connection header received
0929 20:27:30.421299 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:27:30.421309 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:27:30.421565 (+ 256us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:27:30.422708 (+ 1143us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:27:30.423132 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:27:30.424453 (+ 1321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:27:30.427338 (+ 2885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:27:30.428685 (+ 1347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:27:30.429409 (+ 724us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:27:30.429644 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0929 20:27:30.429740 (+ 96us) server_negotiation.cc:299] Negotiation successful
0929 20:27:30.429927 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1383,"thread_start_us":1231,"threads_started":1}
I20260929 20:27:30.430274 3946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:27:30.418527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.67:36899 (local address 127.0.0.1:48412)
0929 20:27:30.420931 (+ 2404us) negotiation.cc:107] Waiting for socket to connect
0929 20:27:30.420953 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 20:27:30.421069 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:27:30.421802 (+ 733us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:27:30.421809 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:27:30.421848 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:27:30.422957 (+ 1109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:27:30.422967 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:27:30.424588 (+ 1621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:27:30.424594 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:27:30.427179 (+ 2585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:27:30.427195 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:27:30.428917 (+ 1722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:27:30.428924 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:27:30.429133 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:27:30.429179 (+ 46us) client_negotiation.cc:769] Sending connection context
0929 20:27:30.429249 (+ 70us) client_negotiation.cc:240] Negotiation successful
0929 20:27:30.429523 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2249,"mutex_wait_us":154,"thread_start_us":2041,"threads_started":1}
I20260929 20:27:30.431046 3948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:27:30.420954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.66:38161 (local address 127.0.0.1:45982)
0929 20:27:30.422760 (+ 1806us) negotiation.cc:107] Waiting for socket to connect
0929 20:27:30.422784 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0929 20:27:30.422944 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:27:30.423582 (+ 638us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:27:30.423587 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:27:30.423611 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:27:30.423922 (+ 311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:27:30.423931 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:27:30.425839 (+ 1908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:27:30.425847 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:27:30.427804 (+ 1957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:27:30.427815 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:27:30.429209 (+ 1394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:27:30.429217 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:27:30.430342 (+ 1125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:27:30.430372 (+ 30us) client_negotiation.cc:769] Sending connection context
0929 20:27:30.430453 (+ 81us) client_negotiation.cc:240] Negotiation successful
0929 20:27:30.430614 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1675,"mutex_wait_us":166,"thread_start_us":618,"threads_started":1}
I20260929 20:27:30.431737 3949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:27:30.421421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45982 (local address 127.17.53.66:38161)
0929 20:27:30.422711 (+ 1290us) server_negotiation.cc:206] Beginning negotiation
0929 20:27:30.422720 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 20:27:30.423026 (+ 306us) server_negotiation.cc:407] Connection header received
0929 20:27:30.423132 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:27:30.423141 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:27:30.423378 (+ 237us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:27:30.423570 (+ 192us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:27:30.424079 (+ 509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:27:30.425661 (+ 1582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:27:30.427949 (+ 2288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:27:30.429019 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:27:30.429154 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:27:30.429354 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0929 20:27:30.431227 (+ 1873us) server_negotiation.cc:299] Negotiation successful
0929 20:27:30.431404 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1137,"thread_start_us":1027,"threads_started":1}
I20260929 20:27:30.431798 3945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:27:30.418511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52162 (local address 127.17.53.65:36159)
0929 20:27:30.421223 (+ 2712us) server_negotiation.cc:206] Beginning negotiation
0929 20:27:30.421232 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 20:27:30.421261 (+ 29us) server_negotiation.cc:407] Connection header received
0929 20:27:30.421363 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:27:30.421370 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:27:30.421629 (+ 259us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:27:30.422055 (+ 426us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:27:30.422810 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:27:30.424350 (+ 1540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:27:30.427077 (+ 2727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:27:30.428581 (+ 1504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:27:30.428737 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:27:30.429005 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0929 20:27:30.431336 (+ 2331us) server_negotiation.cc:299] Negotiation successful
0929 20:27:30.431498 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2545,"thread_start_us":2414,"threads_started":1}
I20260929 20:27:30.434543 3867 tablet_service.cc:1467] Processing CreateTablet for tablet f199a20856184bddb9bc02772653633c (DEFAULT_TABLE table=test-table [id=2aa2a9464e0749bd9e5039270b5bb654]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:27:30.435132 3732 tablet_service.cc:1467] Processing CreateTablet for tablet f199a20856184bddb9bc02772653633c (DEFAULT_TABLE table=test-table [id=2aa2a9464e0749bd9e5039270b5bb654]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:27:30.436172 3867 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f199a20856184bddb9bc02772653633c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:27:30.436559 3732 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f199a20856184bddb9bc02772653633c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:27:30.436088 3596 tablet_service.cc:1467] Processing CreateTablet for tablet f199a20856184bddb9bc02772653633c (DEFAULT_TABLE table=test-table [id=2aa2a9464e0749bd9e5039270b5bb654]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:27:30.437419 3596 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f199a20856184bddb9bc02772653633c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:27:30.447216 3950 tablet_bootstrap.cc:492] T f199a20856184bddb9bc02772653633c P 1da255aea18549f790edb7521f209ec8: Bootstrap starting.
I20260929 20:27:30.451327 3951 tablet_bootstrap.cc:492] T f199a20856184bddb9bc02772653633c P d8c6f447caae49bea30f52d56902fe5b: Bootstrap starting.
I20260929 20:27:30.451752 3950 tablet_bootstrap.cc:654] T f199a20856184bddb9bc02772653633c P 1da255aea18549f790edb7521f209ec8: Neither blocks nor log segments found. Creating new log.
I20260929 20:27:30.453050 3952 tablet_bootstrap.cc:492] T f199a20856184bddb9bc02772653633c P 33b49bd6e11e4391b4928e0861f91f59: Bootstrap starting.
I20260929 20:27:30.453233 3950 log.cc:824] T f199a20856184bddb9bc02772653633c P 1da255aea18549f790edb7521f209ec8: Log is configured to *not* fsync() on all Append() calls
I20260929 20:27:30.455119 3951 tablet_bootstrap.cc:654] T f199a20856184bddb9bc02772653633c P d8c6f447caae49bea30f52d56902fe5b: Neither blocks nor log segments found. Creating new log.
I20260929 20:27:30.456055 3951 log.cc:824] T f199a20856184bddb9bc02772653633c P d8c6f447caae49bea30f52d56902fe5b: Log is configured to *not* fsync() on all Append() calls
I20260929 20:27:30.456568 3952 tablet_bootstrap.cc:654] T f199a20856184bddb9bc02772653633c P 33b49bd6e11e4391b4928e0861f91f59: Neither blocks nor log segments found. Creating new log.
I20260929 20:27:30.456894 3950 tablet_bootstrap.cc:492] T f199a20856184bddb9bc02772653633c P 1da255aea18549f790edb7521f209ec8: No bootstrap required, opened a new log
I20260929 20:27:30.457182 3950 ts_tablet_manager.cc:1397] T f199a20856184bddb9bc02772653633c P 1da255aea18549f790edb7521f209ec8: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.003s
I20260929 20:27:30.457746 3952 log.cc:824] T f199a20856184bddb9bc02772653633c P 33b49bd6e11e4391b4928e0861f91f59: Log is configured to *not* fsync() on all Append() calls
I20260929 20:27:30.459574 3951 tablet_bootstrap.cc:492] T f199a20856184bddb9bc02772653633c P d8c6f447caae49bea30f52d56902fe5b: No bootstrap required, opened a new log
I20260929 20:27:30.459832 3951 ts_tablet_manager.cc:1397] T f199a20856184bddb9bc02772653633c P d8c6f447caae49bea30f52d56902fe5b: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260929 20:27:30.460664 3952 tablet_bootstrap.cc:492] T f199a20856184bddb9bc02772653633c P 33b49bd6e11e4391b4928e0861f91f59: No bootstrap required, opened a new log
I20260929 20:27:30.460916 3952 ts_tablet_manager.cc:1397] T f199a20856184bddb9bc02772653633c P 33b49bd6e11e4391b4928e0861f91f59: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.006s
I20260929 20:27:30.466581 3951 raft_consensus.cc:348] T f199a20856184bddb9bc02772653633c P d8c6f447caae49bea30f52d56902fe5b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33b49bd6e11e4391b4928e0861f91f59" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 36159 } } peers { permanent_uuid: "1da255aea18549f790edb7521f209ec8" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 36899 } } peers { permanent_uuid: "d8c6f447caae49bea30f52d56902fe5b" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 38161 } }
I20260929 20:27:30.466984 3951 raft_consensus.cc:374] T f199a20856184bddb9bc02772653633c P d8c6f447caae49bea30f52d56902fe5b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:27:30.467087 3951 raft_consensus.cc:729] T f199a20856184bddb9bc02772653633c P d8c6f447caae49bea30f52d56902fe5b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8c6f447caae49bea30f52d56902fe5b, State: Initialized, Role: FOLLOWER
I20260929 20:27:30.467767 3951 consensus_queue.cc:260] T f199a20856184bddb9bc02772653633c P d8c6f447caae49bea30f52d56902fe5b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33b49bd6e11e4391b4928e0861f91f59" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 36159 } } peers { permanent_uuid: "1da255aea18549f790edb7521f209ec8" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 36899 } } peers { permanent_uuid: "d8c6f447caae49bea30f52d56902fe5b" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 38161 } }
I20260929 20:27:30.467727 3950 raft_consensus.cc:348] T f199a20856184bddb9bc02772653633c P 1da255aea18549f790edb7521f209ec8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33b49bd6e11e4391b4928e0861f91f59" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 36159 } } peers { permanent_uuid: "1da255aea18549f790edb7521f209ec8" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 36899 } } peers { permanent_uuid: "d8c6f447caae49bea30f52d56902fe5b" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 38161 } }
I20260929 20:27:30.468154 3950 raft_consensus.cc:374] T f199a20856184bddb9bc02772653633c P 1da255aea18549f790edb7521f209ec8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:27:30.468240 3950 raft_consensus.cc:729] T f199a20856184bddb9bc02772653633c P 1da255aea18549f790edb7521f209ec8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1da255aea18549f790edb7521f209ec8, State: Initialized, Role: FOLLOWER
I20260929 20:27:30.468952 3950 consensus_queue.cc:260] T f199a20856184bddb9bc02772653633c P 1da255aea18549f790edb7521f209ec8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33b49bd6e11e4391b4928e0861f91f59" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 36159 } } peers { permanent_uuid: "1da255aea18549f790edb7521f209ec8" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 36899 } } peers { permanent_uuid: "d8c6f447caae49bea30f52d56902fe5b" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 38161 } }
I20260929 20:27:30.470711 3798 heartbeater.cc:503] Master 127.17.53.126:39989 was elected leader, sending a full tablet report...
I20260929 20:27:30.470804 3950 ts_tablet_manager.cc:1428] T f199a20856184bddb9bc02772653633c P 1da255aea18549f790edb7521f209ec8: Time spent starting tablet: real 0.014s user 0.005s sys 0.009s
I20260929 20:27:30.471529 3951 ts_tablet_manager.cc:1428] T f199a20856184bddb9bc02772653633c P d8c6f447caae49bea30f52d56902fe5b: Time spent starting tablet: real 0.012s user 0.008s sys 0.004s
I20260929 20:27:30.471551 3952 raft_consensus.cc:348] T f199a20856184bddb9bc02772653633c P 33b49bd6e11e4391b4928e0861f91f59 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33b49bd6e11e4391b4928e0861f91f59" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 36159 } } peers { permanent_uuid: "1da255aea18549f790edb7521f209ec8" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 36899 } } peers { permanent_uuid: "d8c6f447caae49bea30f52d56902fe5b" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 38161 } }
I20260929 20:27:30.471892 3952 raft_consensus.cc:374] T f199a20856184bddb9bc02772653633c P 33b49bd6e11e4391b4928e0861f91f59 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:27:30.471988 3952 raft_consensus.cc:729] T f199a20856184bddb9bc02772653633c P 33b49bd6e11e4391b4928e0861f91f59 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33b49bd6e11e4391b4928e0861f91f59, State: Initialized, Role: FOLLOWER
I20260929 20:27:30.472083 3932 heartbeater.cc:503] Master 127.17.53.126:39989 was elected leader, sending a full tablet report...
I20260929 20:27:30.472591 3952 consensus_queue.cc:260] T f199a20856184bddb9bc02772653633c P 33b49bd6e11e4391b4928e0861f91f59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33b49bd6e11e4391b4928e0861f91f59" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 36159 } } peers { permanent_uuid: "1da255aea18549f790edb7521f209ec8" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 36899 } } peers { permanent_uuid: "d8c6f447caae49bea30f52d56902fe5b" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 38161 } }
I20260929 20:27:30.474220 3952 ts_tablet_manager.cc:1428] T f199a20856184bddb9bc02772653633c P 33b49bd6e11e4391b4928e0861f91f59: Time spent starting tablet: real 0.013s user 0.005s sys 0.009s
I20260929 20:27:30.518426 3956 raft_consensus.cc:482] T f199a20856184bddb9bc02772653633c P d8c6f447caae49bea30f52d56902fe5b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 20:27:30.518736 3956 raft_consensus.cc:504] T f199a20856184bddb9bc02772653633c P d8c6f447caae49bea30f52d56902fe5b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33b49bd6e11e4391b4928e0861f91f59" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 36159 } } peers { permanent_uuid: "1da255aea18549f790edb7521f209ec8" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 36899 } } peers { permanent_uuid: "d8c6f447caae49bea30f52d56902fe5b" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 38161 } }
I20260929 20:27:30.520534 3956 leader_election.cc:288] T f199a20856184bddb9bc02772653633c P d8c6f447caae49bea30f52d56902fe5b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 33b49bd6e11e4391b4928e0861f91f59 (127.17.53.65:36159), 1da255aea18549f790edb7521f209ec8 (127.17.53.67:36899)
I20260929 20:27:30.527761 3945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:27:30.521091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.53.66:44169 (local address 127.17.53.65:36159)
0929 20:27:30.521378 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0929 20:27:30.521386 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 20:27:30.522207 (+ 821us) server_negotiation.cc:407] Connection header received
0929 20:27:30.522427 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:27:30.522433 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:27:30.522500 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:27:30.522598 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:27:30.523241 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:27:30.523983 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:27:30.525982 (+ 1999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:27:30.526778 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:27:30.527013 (+ 235us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:27:30.527087 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0929 20:27:30.527368 (+ 281us) server_negotiation.cc:299] Negotiation successful
0929 20:27:30.527495 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
I20260929 20:27:30.527758 3947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:27:30.521025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.53.66:34543 (local address 127.17.53.67:36899)
0929 20:27:30.521252 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0929 20:27:30.521260 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 20:27:30.522273 (+ 1013us) server_negotiation.cc:407] Connection header received
0929 20:27:30.522483 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:27:30.522489 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:27:30.522555 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:27:30.522648 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:27:30.523338 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:27:30.524091 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:27:30.525968 (+ 1877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:27:30.526735 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:27:30.526944 (+ 209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:27:30.527024 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0929 20:27:30.527369 (+ 345us) server_negotiation.cc:299] Negotiation successful
0929 20:27:30.527509 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260929 20:27:30.527920 3960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:27:30.521036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.67:36899 (local address 127.17.53.66:34543)
0929 20:27:30.522190 (+ 1154us) negotiation.cc:107] Waiting for socket to connect
0929 20:27:30.522205 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0929 20:27:30.522330 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:27:30.522661 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:27:30.522666 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:27:30.522687 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:27:30.523188 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:27:30.523197 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:27:30.524227 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:27:30.524232 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:27:30.525816 (+ 1584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:27:30.525828 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:27:30.526997 (+ 1169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:27:30.527005 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:27:30.527174 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:27:30.527229 (+ 55us) client_negotiation.cc:769] Sending connection context
0929 20:27:30.527339 (+ 110us) client_negotiation.cc:240] Negotiation successful
0929 20:27:30.527478 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1048,"mutex_wait_us":47,"thread_start_us":674,"threads_started":1}
I20260929 20:27:30.528050 3959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:27:30.520967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.65:36159 (local address 127.17.53.66:44169)
0929 20:27:30.522103 (+ 1136us) negotiation.cc:107] Waiting for socket to connect
0929 20:27:30.522119 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 20:27:30.522282 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:27:30.522617 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:27:30.522621 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:27:30.522645 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:27:30.523091 (+ 446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:27:30.523102 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:27:30.524124 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:27:30.524131 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:27:30.525804 (+ 1673us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:27:30.525816 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:27:30.527037 (+ 1221us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:27:30.527044 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:27:30.527220 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:27:30.527242 (+ 22us) client_negotiation.cc:769] Sending connection context
0929 20:27:30.527340 (+ 98us) client_negotiation.cc:240] Negotiation successful
0929 20:27:30.527481 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":993,"mutex_wait_us":5,"thread_start_us":617,"threads_started":1}
I20260929 20:27:30.528983 3616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f199a20856184bddb9bc02772653633c" candidate_uuid: "d8c6f447caae49bea30f52d56902fe5b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33b49bd6e11e4391b4928e0861f91f59" is_pre_election: true
I20260929 20:27:30.529012 3887 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f199a20856184bddb9bc02772653633c" candidate_uuid: "d8c6f447caae49bea30f52d56902fe5b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1da255aea18549f790edb7521f209ec8" is_pre_election: true
I20260929 20:27:30.529495 3887 raft_consensus.cc:2454] T f199a20856184bddb9bc02772653633c P 1da255aea18549f790edb7521f209ec8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8c6f447caae49bea30f52d56902fe5b in term 0.
I20260929 20:27:30.529493 3616 raft_consensus.cc:2454] T f199a20856184bddb9bc02772653633c P 33b49bd6e11e4391b4928e0861f91f59 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8c6f447caae49bea30f52d56902fe5b in term 0.
I20260929 20:27:30.530148 3685 leader_election.cc:302] T f199a20856184bddb9bc02772653633c P d8c6f447caae49bea30f52d56902fe5b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 1da255aea18549f790edb7521f209ec8, 33b49bd6e11e4391b4928e0861f91f59, d8c6f447caae49bea30f52d56902fe5b; no voters:
I20260929 20:27:30.530552 3956 raft_consensus.cc:2781] T f199a20856184bddb9bc02772653633c P d8c6f447caae49bea30f52d56902fe5b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 20:27:30.530702 3956 raft_consensus.cc:482] T f199a20856184bddb9bc02772653633c P d8c6f447caae49bea30f52d56902fe5b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 20:27:30.530834 3956 raft_consensus.cc:3037] T f199a20856184bddb9bc02772653633c P d8c6f447caae49bea30f52d56902fe5b [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:27:30.533077 3956 raft_consensus.cc:504] T f199a20856184bddb9bc02772653633c P d8c6f447caae49bea30f52d56902fe5b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33b49bd6e11e4391b4928e0861f91f59" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 36159 } } peers { permanent_uuid: "1da255aea18549f790edb7521f209ec8" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 36899 } } peers { permanent_uuid: "d8c6f447caae49bea30f52d56902fe5b" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 38161 } }
I20260929 20:27:30.533725 3956 leader_election.cc:288] T f199a20856184bddb9bc02772653633c P d8c6f447caae49bea30f52d56902fe5b [CANDIDATE]: Term 1 election: Requested vote from peers 33b49bd6e11e4391b4928e0861f91f59 (127.17.53.65:36159), 1da255aea18549f790edb7521f209ec8 (127.17.53.67:36899)
I20260929 20:27:30.534116 3616 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f199a20856184bddb9bc02772653633c" candidate_uuid: "d8c6f447caae49bea30f52d56902fe5b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33b49bd6e11e4391b4928e0861f91f59"
I20260929 20:27:30.534201 3887 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f199a20856184bddb9bc02772653633c" candidate_uuid: "d8c6f447caae49bea30f52d56902fe5b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1da255aea18549f790edb7521f209ec8"
I20260929 20:27:30.534349 3616 raft_consensus.cc:3037] T f199a20856184bddb9bc02772653633c P 33b49bd6e11e4391b4928e0861f91f59 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:27:30.534384 3887 raft_consensus.cc:3037] T f199a20856184bddb9bc02772653633c P 1da255aea18549f790edb7521f209ec8 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:27:30.537628 3616 raft_consensus.cc:2454] T f199a20856184bddb9bc02772653633c P 33b49bd6e11e4391b4928e0861f91f59 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8c6f447caae49bea30f52d56902fe5b in term 1.
I20260929 20:27:30.537642 3887 raft_consensus.cc:2454] T f199a20856184bddb9bc02772653633c P 1da255aea18549f790edb7521f209ec8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8c6f447caae49bea30f52d56902fe5b in term 1.
I20260929 20:27:30.538086 3686 leader_election.cc:302] T f199a20856184bddb9bc02772653633c P d8c6f447caae49bea30f52d56902fe5b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1da255aea18549f790edb7521f209ec8, d8c6f447caae49bea30f52d56902fe5b; no voters:
I20260929 20:27:30.538435 3956 raft_consensus.cc:2781] T f199a20856184bddb9bc02772653633c P d8c6f447caae49bea30f52d56902fe5b [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:27:30.538932 3956 raft_consensus.cc:686] T f199a20856184bddb9bc02772653633c P d8c6f447caae49bea30f52d56902fe5b [term 1 LEADER]: Becoming Leader. State: Replica: d8c6f447caae49bea30f52d56902fe5b, State: Running, Role: LEADER
I20260929 20:27:30.539425 3956 consensus_queue.cc:237] T f199a20856184bddb9bc02772653633c P d8c6f447caae49bea30f52d56902fe5b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33b49bd6e11e4391b4928e0861f91f59" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 36159 } } peers { permanent_uuid: "1da255aea18549f790edb7521f209ec8" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 36899 } } peers { permanent_uuid: "d8c6f447caae49bea30f52d56902fe5b" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 38161 } }
I20260929 20:27:30.544361 3472 catalog_manager.cc:5495] T f199a20856184bddb9bc02772653633c P d8c6f447caae49bea30f52d56902fe5b reported cstate change: term changed from 0 to 1, leader changed from <none> to d8c6f447caae49bea30f52d56902fe5b (127.17.53.66). New cstate: current_term: 1 leader_uuid: "d8c6f447caae49bea30f52d56902fe5b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "33b49bd6e11e4391b4928e0861f91f59" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 36159 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1da255aea18549f790edb7521f209ec8" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 36899 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d8c6f447caae49bea30f52d56902fe5b" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 38161 } health_report { overall_health: HEALTHY } } }
W20260929 20:27:30.555038 3663 tablet.cc:2369] T f199a20856184bddb9bc02772653633c P 33b49bd6e11e4391b4928e0861f91f59: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 20:27:30.579924 3799 tablet.cc:2369] T f199a20856184bddb9bc02772653633c P d8c6f447caae49bea30f52d56902fe5b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 20:27:30.585152 3949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:27:30.579542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45996 (local address 127.17.53.66:38161)
0929 20:27:30.579778 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0929 20:27:30.579785 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:27:30.579911 (+ 126us) server_negotiation.cc:407] Connection header received
0929 20:27:30.580165 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:27:30.580170 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:27:30.580310 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:27:30.580472 (+ 162us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 20:27:30.580819 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:27:30.581547 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:27:30.582948 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:27:30.583365 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:27:30.583460 (+ 95us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:27:30.584002 (+ 542us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 20:27:30.584485 (+ 483us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 20:27:30.584607 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0929 20:27:30.584856 (+ 249us) server_negotiation.cc:299] Negotiation successful
0929 20:27:30.584916 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 20:27:30.593251 3887 raft_consensus.cc:1261] T f199a20856184bddb9bc02772653633c P 1da255aea18549f790edb7521f209ec8 [term 1 FOLLOWER]: Refusing update from remote peer d8c6f447caae49bea30f52d56902fe5b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:27:30.593286 3616 raft_consensus.cc:1261] T f199a20856184bddb9bc02772653633c P 33b49bd6e11e4391b4928e0861f91f59 [term 1 FOLLOWER]: Refusing update from remote peer d8c6f447caae49bea30f52d56902fe5b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260929 20:27:30.594080 3933 tablet.cc:2369] T f199a20856184bddb9bc02772653633c P 1da255aea18549f790edb7521f209ec8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 20:27:30.594202 3961 consensus_queue.cc:1035] T f199a20856184bddb9bc02772653633c P d8c6f447caae49bea30f52d56902fe5b [LEADER]: Connected to new peer: Peer: permanent_uuid: "33b49bd6e11e4391b4928e0861f91f59" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 36159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:27:30.594614 3956 consensus_queue.cc:1035] T f199a20856184bddb9bc02772653633c P d8c6f447caae49bea30f52d56902fe5b [LEADER]: Connected to new peer: Peer: permanent_uuid: "1da255aea18549f790edb7521f209ec8" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 36899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:27:30.606273 3964 mvcc.cc:205] Tried to move back new op lower bound from 7334763112819916800 to 7334763112613834752. Current Snapshot: MvccSnapshot[applied={T|T < 7334763112819916800}]
I20260929 20:27:30.607436 3965 mvcc.cc:205] Tried to move back new op lower bound from 7334763112819916800 to 7334763112613834752. Current Snapshot: MvccSnapshot[applied={T|T < 7334763112819916800}]
I20260929 20:27:30.621043 3472 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40756:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 20:27:30.621315 3472 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40756:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 20:27:30.624830 3472 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 52ae69cb81aa4706bf6f6f1dd771c948: Sending DeleteTablet for 3 replicas of tablet f199a20856184bddb9bc02772653633c
I20260929 20:27:30.626063 3596 tablet_service.cc:1514] Processing DeleteTablet for tablet f199a20856184bddb9bc02772653633c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:27:30 UTC) from {username='kudu', principal='kudu/127.17.53.126@KRBTEST.COM'} at 127.0.0.1:52162
I20260929 20:27:30.626116 3867 tablet_service.cc:1514] Processing DeleteTablet for tablet f199a20856184bddb9bc02772653633c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:27:30 UTC) from {username='kudu', principal='kudu/127.17.53.126@KRBTEST.COM'} at 127.0.0.1:48412
I20260929 20:27:30.626268 3732 tablet_service.cc:1514] Processing DeleteTablet for tablet f199a20856184bddb9bc02772653633c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:27:30 UTC) from {username='kudu', principal='kudu/127.17.53.126@KRBTEST.COM'} at 127.0.0.1:45982
I20260929 20:27:30.629086 17621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu pid 3533
I20260929 20:27:30.643764 3976 tablet_replica.cc:330] T f199a20856184bddb9bc02772653633c P d8c6f447caae49bea30f52d56902fe5b: stopping tablet replica
I20260929 20:27:30.644299 3976 raft_consensus.cc:2229] T f199a20856184bddb9bc02772653633c P d8c6f447caae49bea30f52d56902fe5b [term 1 LEADER]: Raft consensus shutting down.
I20260929 20:27:30.645018 3976 raft_consensus.cc:2258] T f199a20856184bddb9bc02772653633c P d8c6f447caae49bea30f52d56902fe5b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:27:30.648741 3976 ts_tablet_manager.cc:1905] T f199a20856184bddb9bc02772653633c P d8c6f447caae49bea30f52d56902fe5b: Deleting tablet data with delete state TABLET_DATA_DELETED
Sep 29 20:27:30 dist-test-slave-zbnc krb5kdc[3427](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713648, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.53.65@KRBTEST.COM
I20260929 20:27:30.655386 3974 tablet_replica.cc:330] T f199a20856184bddb9bc02772653633c P 1da255aea18549f790edb7521f209ec8: stopping tablet replica
I20260929 20:27:30.655884 3974 raft_consensus.cc:2229] T f199a20856184bddb9bc02772653633c P 1da255aea18549f790edb7521f209ec8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 20:27:30.656366 3974 raft_consensus.cc:2258] T f199a20856184bddb9bc02772653633c P 1da255aea18549f790edb7521f209ec8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:27:30.657090 3975 tablet_replica.cc:330] T f199a20856184bddb9bc02772653633c P 33b49bd6e11e4391b4928e0861f91f59: stopping tablet replica
I20260929 20:27:30.657454 3975 raft_consensus.cc:2229] T f199a20856184bddb9bc02772653633c P 33b49bd6e11e4391b4928e0861f91f59 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 20:27:30.657862 3975 raft_consensus.cc:2258] T f199a20856184bddb9bc02772653633c P 33b49bd6e11e4391b4928e0861f91f59 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:27:30.660168 3974 ts_tablet_manager.cc:1905] T f199a20856184bddb9bc02772653633c P 1da255aea18549f790edb7521f209ec8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:27:30.663880 3976 ts_tablet_manager.cc:1918] T f199a20856184bddb9bc02772653633c P d8c6f447caae49bea30f52d56902fe5b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 20:27:30.664062 3976 log.cc:1196] T f199a20856184bddb9bc02772653633c P d8c6f447caae49bea30f52d56902fe5b: Deleting WAL directory at /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790713339161492-17621-0/minicluster-data/ts-1/wal/wals/f199a20856184bddb9bc02772653633c
I20260929 20:27:30.664675 3976 ts_tablet_manager.cc:1939] T f199a20856184bddb9bc02772653633c P d8c6f447caae49bea30f52d56902fe5b: Deleting consensus metadata
I20260929 20:27:30.665737 3975 ts_tablet_manager.cc:1905] T f199a20856184bddb9bc02772653633c P 33b49bd6e11e4391b4928e0861f91f59: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:27:30.666652 3457 catalog_manager.cc:4841] TS d8c6f447caae49bea30f52d56902fe5b (127.17.53.66:38161): tablet f199a20856184bddb9bc02772653633c (table test-table [id=2aa2a9464e0749bd9e5039270b5bb654]) successfully deleted
I20260929 20:27:30.667277 3974 ts_tablet_manager.cc:1918] T f199a20856184bddb9bc02772653633c P 1da255aea18549f790edb7521f209ec8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 20:27:30.667450 3974 log.cc:1196] T f199a20856184bddb9bc02772653633c P 1da255aea18549f790edb7521f209ec8: Deleting WAL directory at /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790713339161492-17621-0/minicluster-data/ts-2/wal/wals/f199a20856184bddb9bc02772653633c
I20260929 20:27:30.667982 3974 ts_tablet_manager.cc:1939] T f199a20856184bddb9bc02772653633c P 1da255aea18549f790edb7521f209ec8: Deleting consensus metadata
I20260929 20:27:30.669775 3457 catalog_manager.cc:4841] TS 1da255aea18549f790edb7521f209ec8 (127.17.53.67:36899): tablet f199a20856184bddb9bc02772653633c (table test-table [id=2aa2a9464e0749bd9e5039270b5bb654]) successfully deleted
I20260929 20:27:30.672992 3975 ts_tablet_manager.cc:1918] T f199a20856184bddb9bc02772653633c P 33b49bd6e11e4391b4928e0861f91f59: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 20:27:30.673157 3975 log.cc:1196] T f199a20856184bddb9bc02772653633c P 33b49bd6e11e4391b4928e0861f91f59: Deleting WAL directory at /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790713339161492-17621-0/minicluster-data/ts-0/wal/wals/f199a20856184bddb9bc02772653633c
I20260929 20:27:30.673647 3975 ts_tablet_manager.cc:1939] T f199a20856184bddb9bc02772653633c P 33b49bd6e11e4391b4928e0861f91f59: Deleting consensus metadata
I20260929 20:27:30.675393 3456 catalog_manager.cc:4841] TS 33b49bd6e11e4391b4928e0861f91f59 (127.17.53.65:36159): tablet f199a20856184bddb9bc02772653633c (table test-table [id=2aa2a9464e0749bd9e5039270b5bb654]) successfully deleted
I20260929 20:27:30.682658 3945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:27:30.630244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52178 (local address 127.17.53.65:36159)
0929 20:27:30.630433 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0929 20:27:30.630442 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 20:27:30.641690 (+ 11248us) server_negotiation.cc:407] Connection header received
0929 20:27:30.641908 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:27:30.641917 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:27:30.642044 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:27:30.642165 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:27:30.644669 (+ 2504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:27:30.645883 (+ 1214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:27:30.647939 (+ 2056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:27:30.648640 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:27:30.648760 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:27:30.675346 (+ 26586us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:27:30.675374 (+ 28us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:27:30.675392 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:27:30.675447 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:27:30.679308 (+ 3861us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:27:30.680279 (+ 971us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:27:30.680290 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:27:30.680294 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:27:30.680470 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:27:30.680922 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:27:30.680928 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:27:30.680931 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:27:30.681405 (+ 474us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:27:30.681703 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0929 20:27:30.682041 (+ 338us) server_negotiation.cc:299] Negotiation successful
0929 20:27:30.682267 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260929 20:27:30.683323 3657 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:27:30.773768 17621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu with pid 3533
I20260929 20:27:30.791807 17621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu pid 3669
Sep 29 20:27:30 dist-test-slave-zbnc krb5kdc[3427](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713648, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.53.66@KRBTEST.COM
I20260929 20:27:30.809057 3949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:27:30.792602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46002 (local address 127.17.53.66:38161)
0929 20:27:30.792795 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0929 20:27:30.792801 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:27:30.792821 (+ 20us) server_negotiation.cc:407] Connection header received
0929 20:27:30.792994 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:27:30.793000 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:27:30.793111 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:27:30.793226 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:27:30.794483 (+ 1257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:27:30.795749 (+ 1266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:27:30.796992 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:27:30.797662 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:27:30.797786 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:27:30.802600 (+ 4814us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:27:30.802615 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:27:30.802644 (+ 29us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:27:30.802700 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:27:30.805813 (+ 3113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:27:30.806646 (+ 833us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:27:30.806659 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:27:30.806661 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:27:30.806810 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:27:30.807416 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:27:30.807419 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:27:30.807421 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:27:30.807811 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:27:30.808059 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0929 20:27:30.808396 (+ 337us) server_negotiation.cc:299] Negotiation successful
0929 20:27:30.808588 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260929 20:27:30.809827 3793 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:27:30.925529 17621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu with pid 3669
I20260929 20:27:30.945516 17621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu pid 3804
Sep 29 20:27:30 dist-test-slave-zbnc krb5kdc[3427](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713648, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.53.67@KRBTEST.COM
I20260929 20:27:30.960687 3947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:27:30.946346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48422 (local address 127.17.53.67:36899)
0929 20:27:30.946532 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0929 20:27:30.946540 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 20:27:30.946559 (+ 19us) server_negotiation.cc:407] Connection header received
0929 20:27:30.946753 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:27:30.946759 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:27:30.946846 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:27:30.946984 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:27:30.948325 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:27:30.949062 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:27:30.950325 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:27:30.950721 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:27:30.950874 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:27:30.955302 (+ 4428us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:27:30.955316 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:27:30.955331 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:27:30.955363 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:27:30.957884 (+ 2521us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:27:30.958600 (+ 716us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:27:30.958608 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:27:30.958610 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:27:30.958744 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:27:30.959146 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:27:30.959151 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:27:30.959162 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:27:30.959542 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:27:30.959827 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0929 20:27:30.960135 (+ 308us) server_negotiation.cc:299] Negotiation successful
0929 20:27:30.960373 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 20:27:30.961318 3927 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:27:31.050796 17621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu with pid 3804
I20260929 20:27:31.072158 17621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu pid 3440
I20260929 20:27:31.073325 3502 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:27:31.184195 17621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu with pid 3440
2026-09-29T20:27:31Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (4348 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-29T20:27:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T20:27:31Z Disabled control of system clock
I20260929 20:27:31.262573 17621 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
/tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790713339161492-17621-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790713339161492-17621-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790713339161492-17621-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790713339161492-17621-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.53.126:39255
--webserver_interface=127.17.53.126
--webserver_port=0
--builtin_ntp_servers=127.17.53.84:34029
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.53.126:39255
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 20:27:31.600315 3985 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:27:31.600615 3985 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:27:31.600695 3985 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:27:31.608012 3985 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:27:31.608124 3985 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:27:31.608162 3985 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:27:31.608211 3985 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:27:31.608258 3985 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:27:31.617565 3985 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.53.84:34029
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790713339161492-17621-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790713339161492-17621-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.53.126:39255
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.53.126:39255
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790713339161492-17621-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.53.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790713339161492-17621-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:27:31.618844 3985 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:27:31.620637 3985 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:27:31.632407 3991 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:27:31.632598 3990 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:27:31.632601 3993 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:27:31.633158 3985 server_base.cc:1034] running on GCE node
I20260929 20:27:31.634066 3985 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:27:31.635560 3985 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:27:31.636770 3985 hybrid_clock.cc:648] HybridClock initialized: now 1790713651636721 us; error 34 us; skew 500 ppm
I20260929 20:27:31.637148 3985 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 20:27:31.639523 3985 webserver.cc:460] Webserver started at http://127.17.53.126:43263/ using document root <none> and password file <none>
I20260929 20:27:31.640230 3985 fs_manager.cc:359] Metadata directory not provided
I20260929 20:27:31.640355 3985 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:27:31.640683 3985 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:27:31.643713 3985 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790713339161492-17621-0/minicluster-data/master-0/data/instance:
uuid: "b046fe18e38a456db218e661faed4a28"
format_stamp: "Formatted at 2026-09-29 20:27:31 on dist-test-slave-zbnc"
server_key: "b60e74a8027b51bbea3abbc812fed36d"
server_key_iv: "e687cb36483081352de68390006afd3b"
server_key_version: "encryptionkey@0"
I20260929 20:27:31.644536 3985 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790713339161492-17621-0/minicluster-data/master-0/wal/instance:
uuid: "b046fe18e38a456db218e661faed4a28"
format_stamp: "Formatted at 2026-09-29 20:27:31 on dist-test-slave-zbnc"
server_key: "b60e74a8027b51bbea3abbc812fed36d"
server_key_iv: "e687cb36483081352de68390006afd3b"
server_key_version: "encryptionkey@0"
I20260929 20:27:31.649699 3985 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 20:27:31.653169 3999 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:27:31.654539 3985 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 20:27:31.654726 3985 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790713339161492-17621-0/minicluster-data/master-0/data,/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790713339161492-17621-0/minicluster-data/master-0/wal
uuid: "b046fe18e38a456db218e661faed4a28"
format_stamp: "Formatted at 2026-09-29 20:27:31 on dist-test-slave-zbnc"
server_key: "b60e74a8027b51bbea3abbc812fed36d"
server_key_iv: "e687cb36483081352de68390006afd3b"
server_key_version: "encryptionkey@0"
I20260929 20:27:31.654985 3985 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790713339161492-17621-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790713339161492-17621-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790713339161492-17621-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:27:31.680721 3985 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:27:31.681635 3985 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:27:31.681928 3985 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:27:31.700062 3985 rpc_server.cc:307] RPC server started. Bound to: 127.17.53.126:39255
I20260929 20:27:31.700071 4050 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.53.126:39255 every 8 connection(s)
I20260929 20:27:31.701449 3985 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790713339161492-17621-0/minicluster-data/master-0/data/info.pb
I20260929 20:27:31.705323 4051 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:27:31.706394 17621 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu as pid 3985
I20260929 20:27:31.706616 17621 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790713339161492-17621-0/minicluster-data/master-0/wal/instance
I20260929 20:27:31.707262 17621 external_mini_cluster.cc:1428] Setting key 9c245e8228517b91c01091e238d4f947
I20260929 20:27:31.719769 4051 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b046fe18e38a456db218e661faed4a28: Bootstrap starting.
I20260929 20:27:31.723109 4054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:27:31.709403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52522 (local address 127.17.53.126:39255)
0929 20:27:31.710560 (+ 1157us) server_negotiation.cc:206] Beginning negotiation
0929 20:27:31.710579 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0929 20:27:31.710636 (+ 57us) server_negotiation.cc:407] Connection header received
0929 20:27:31.712352 (+ 1716us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:27:31.712385 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:27:31.713037 (+ 652us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:27:31.713473 (+ 436us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:27:31.714067 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:27:31.716023 (+ 1956us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:27:31.717623 (+ 1600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:27:31.718662 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:27:31.718834 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:27:31.720172 (+ 1338us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:27:31.720182 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:27:31.720200 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:27:31.720901 (+ 701us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:27:31.721022 (+ 121us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:27:31.721049 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:27:31.721164 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0929 20:27:31.721405 (+ 241us) server_negotiation.cc:299] Negotiation successful
0929 20:27:31.721801 (+ 396us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":626,"thread_start_us":394,"threads_started":1}
I20260929 20:27:31.723745 4051 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b046fe18e38a456db218e661faed4a28: Neither blocks nor log segments found. Creating new log.
I20260929 20:27:31.724763 4051 log.cc:824] T 00000000000000000000000000000000 P b046fe18e38a456db218e661faed4a28: Log is configured to *not* fsync() on all Append() calls
I20260929 20:27:31.727560 4051 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b046fe18e38a456db218e661faed4a28: No bootstrap required, opened a new log
I20260929 20:27:31.735172 4051 raft_consensus.cc:348] T 00000000000000000000000000000000 P b046fe18e38a456db218e661faed4a28 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b046fe18e38a456db218e661faed4a28" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 39255 } }
I20260929 20:27:31.735577 4051 raft_consensus.cc:374] T 00000000000000000000000000000000 P b046fe18e38a456db218e661faed4a28 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:27:31.735680 4051 raft_consensus.cc:729] T 00000000000000000000000000000000 P b046fe18e38a456db218e661faed4a28 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b046fe18e38a456db218e661faed4a28, State: Initialized, Role: FOLLOWER
I20260929 20:27:31.736255 4051 consensus_queue.cc:260] T 00000000000000000000000000000000 P b046fe18e38a456db218e661faed4a28 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b046fe18e38a456db218e661faed4a28" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 39255 } }
I20260929 20:27:31.736505 4051 raft_consensus.cc:388] T 00000000000000000000000000000000 P b046fe18e38a456db218e661faed4a28 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 20:27:31.736636 4051 raft_consensus.cc:482] T 00000000000000000000000000000000 P b046fe18e38a456db218e661faed4a28 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 20:27:31.736799 4051 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b046fe18e38a456db218e661faed4a28 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:27:31.739217 4051 raft_consensus.cc:504] T 00000000000000000000000000000000 P b046fe18e38a456db218e661faed4a28 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b046fe18e38a456db218e661faed4a28" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 39255 } }
I20260929 20:27:31.739749 4051 leader_election.cc:302] T 00000000000000000000000000000000 P b046fe18e38a456db218e661faed4a28 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b046fe18e38a456db218e661faed4a28; no voters:
I20260929 20:27:31.740384 4051 leader_election.cc:288] T 00000000000000000000000000000000 P b046fe18e38a456db218e661faed4a28 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 20:27:31.740571 4056 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b046fe18e38a456db218e661faed4a28 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:27:31.741474 4056 raft_consensus.cc:686] T 00000000000000000000000000000000 P b046fe18e38a456db218e661faed4a28 [term 1 LEADER]: Becoming Leader. State: Replica: b046fe18e38a456db218e661faed4a28, State: Running, Role: LEADER
I20260929 20:27:31.742118 4056 consensus_queue.cc:237] T 00000000000000000000000000000000 P b046fe18e38a456db218e661faed4a28 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b046fe18e38a456db218e661faed4a28" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 39255 } }
I20260929 20:27:31.742710 4051 sys_catalog.cc:564] T 00000000000000000000000000000000 P b046fe18e38a456db218e661faed4a28 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:27:31.745421 4058 sys_catalog.cc:455] T 00000000000000000000000000000000 P b046fe18e38a456db218e661faed4a28 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b046fe18e38a456db218e661faed4a28. Latest consensus state: current_term: 1 leader_uuid: "b046fe18e38a456db218e661faed4a28" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b046fe18e38a456db218e661faed4a28" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 39255 } } }
I20260929 20:27:31.745889 4058 sys_catalog.cc:458] T 00000000000000000000000000000000 P b046fe18e38a456db218e661faed4a28 [sys.catalog]: This master's current role is: LEADER
I20260929 20:27:31.746584 4057 sys_catalog.cc:455] T 00000000000000000000000000000000 P b046fe18e38a456db218e661faed4a28 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b046fe18e38a456db218e661faed4a28" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b046fe18e38a456db218e661faed4a28" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 39255 } } }
I20260929 20:27:31.747108 4057 sys_catalog.cc:458] T 00000000000000000000000000000000 P b046fe18e38a456db218e661faed4a28 [sys.catalog]: This master's current role is: LEADER
I20260929 20:27:31.755568 4065 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:27:31.760527 4065 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:27:31.769248 4065 catalog_manager.cc:1347] Generated new cluster ID: 214064001bec4fb49726dc6a629c368d
I20260929 20:27:31.769375 4065 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:27:31.817637 4065 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 20:27:31.819036 4065 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:27:31.828047 4065 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b046fe18e38a456db218e661faed4a28: Generated new TSK 0
I20260929 20:27:31.828866 4065 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 20:27:31.840660 17621 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
/tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790713339161492-17621-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790713339161492-17621-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790713339161492-17621-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790713339161492-17621-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.53.65:0
--local_ip_for_outbound_sockets=127.17.53.65
--webserver_interface=127.17.53.65
--webserver_port=0
--tserver_master_addrs=127.17.53.126:39255
--builtin_ntp_servers=127.17.53.84:34029
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 20:27:32.193816 4075 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:27:32.194096 4075 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:27:32.194258 4075 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:27:32.201624 4075 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:27:32.201725 4075 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:27:32.201937 4075 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.53.65
I20260929 20:27:32.211189 4075 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.53.84:34029
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790713339161492-17621-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790713339161492-17621-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.53.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790713339161492-17621-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.17.53.65
--webserver_port=0
--tserver_master_addrs=127.17.53.126:39255
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.53.65
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790713339161492-17621-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:27:32.212497 4075 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:27:32.214270 4075 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:27:32.228037 4080 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:27:32.228255 4081 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:27:32.228294 4083 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:27:32.229285 4075 server_base.cc:1034] running on GCE node
I20260929 20:27:32.229894 4075 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:27:32.230892 4075 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:27:32.232155 4075 hybrid_clock.cc:648] HybridClock initialized: now 1790713652232098 us; error 48 us; skew 500 ppm
I20260929 20:27:32.232540 4075 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 20:27:32.234927 4075 webserver.cc:460] Webserver started at http://127.17.53.65:43073/ using document root <none> and password file <none>
I20260929 20:27:32.235636 4075 fs_manager.cc:359] Metadata directory not provided
I20260929 20:27:32.235770 4075 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:27:32.236088 4075 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:27:32.239096 4075 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790713339161492-17621-0/minicluster-data/ts-0/data/instance:
uuid: "2eb6300c78c4445e9c3d409a041c1a56"
format_stamp: "Formatted at 2026-09-29 20:27:32 on dist-test-slave-zbnc"
server_key: "f2c8ae5324901514cc69ba2095ab8bcd"
server_key_iv: "644e84514496accab3f07d51ecc9e25a"
server_key_version: "encryptionkey@0"
I20260929 20:27:32.239898 4075 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790713339161492-17621-0/minicluster-data/ts-0/wal/instance:
uuid: "2eb6300c78c4445e9c3d409a041c1a56"
format_stamp: "Formatted at 2026-09-29 20:27:32 on dist-test-slave-zbnc"
server_key: "f2c8ae5324901514cc69ba2095ab8bcd"
server_key_iv: "644e84514496accab3f07d51ecc9e25a"
server_key_version: "encryptionkey@0"
I20260929 20:27:32.245268 4075 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 20:27:32.248950 4089 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:27:32.250293 4075 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 20:27:32.250471 4075 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790713339161492-17621-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790713339161492-17621-0/minicluster-data/ts-0/wal
uuid: "2eb6300c78c4445e9c3d409a041c1a56"
format_stamp: "Formatted at 2026-09-29 20:27:32 on dist-test-slave-zbnc"
server_key: "f2c8ae5324901514cc69ba2095ab8bcd"
server_key_iv: "644e84514496accab3f07d51ecc9e25a"
server_key_version: "encryptionkey@0"
I20260929 20:27:32.250705 4075 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790713339161492-17621-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790713339161492-17621-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790713339161492-17621-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:27:32.269012 4075 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:27:32.269821 4075 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:27:32.270133 4075 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:27:32.271147 4075 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:27:32.272617 4075 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:27:32.272737 4075 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:27:32.272861 4075 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:27:32.272955 4075 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:27:32.306381 4075 rpc_server.cc:307] RPC server started. Bound to: 127.17.53.65:45935
I20260929 20:27:32.306438 4202 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.53.65:45935 every 8 connection(s)
I20260929 20:27:32.307837 4075 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790713339161492-17621-0/minicluster-data/ts-0/data/info.pb
I20260929 20:27:32.309759 17621 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu as pid 4075
I20260929 20:27:32.309938 17621 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790713339161492-17621-0/minicluster-data/ts-0/wal/instance
I20260929 20:27:32.310406 17621 external_mini_cluster.cc:1428] Setting key d8e284790eba3f3ee643900abf81a1e7
I20260929 20:27:32.313443 17621 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
/tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790713339161492-17621-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790713339161492-17621-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790713339161492-17621-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790713339161492-17621-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.53.66:0
--local_ip_for_outbound_sockets=127.17.53.66
--webserver_interface=127.17.53.66
--webserver_port=0
--tserver_master_addrs=127.17.53.126:39255
--builtin_ntp_servers=127.17.53.84:34029
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 20:27:32.322232 4205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:27:32.310659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.53.65:37817 (local address 127.17.53.126:39255)
0929 20:27:32.311207 (+ 548us) server_negotiation.cc:206] Beginning negotiation
0929 20:27:32.311212 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:27:32.312799 (+ 1587us) server_negotiation.cc:407] Connection header received
0929 20:27:32.314661 (+ 1862us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:27:32.314670 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:27:32.314762 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:27:32.314918 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:27:32.316007 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:27:32.317173 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:27:32.318717 (+ 1544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:27:32.319462 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:27:32.319605 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:27:32.321194 (+ 1589us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:27:32.321203 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:27:32.321210 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:27:32.321238 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:27:32.321303 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:27:32.321324 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:27:32.321422 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0929 20:27:32.321858 (+ 436us) server_negotiation.cc:299] Negotiation successful
0929 20:27:32.321948 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":405,"thread_start_us":282,"threads_started":1}
I20260929 20:27:32.323378 4206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:27:32.311481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:39255 (local address 127.17.53.65:37817)
0929 20:27:32.312628 (+ 1147us) negotiation.cc:107] Waiting for socket to connect
0929 20:27:32.312682 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0929 20:27:32.314394 (+ 1712us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:27:32.315155 (+ 761us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:27:32.315164 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:27:32.315251 (+ 87us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:27:32.315841 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:27:32.315857 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:27:32.317330 (+ 1473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:27:32.317334 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:27:32.318577 (+ 1243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:27:32.318589 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:27:32.319678 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:27:32.319686 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:27:32.319983 (+ 297us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:27:32.320813 (+ 830us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 20:27:32.320819 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:27:32.320868 (+ 49us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 20:27:32.320964 (+ 96us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:27:32.321448 (+ 484us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:27:32.321462 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:27:32.321472 (+ 10us) client_negotiation.cc:769] Sending connection context
0929 20:27:32.321709 (+ 237us) client_negotiation.cc:240] Negotiation successful
0929 20:27:32.322164 (+ 455us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":564,"thread_start_us":370,"threads_started":1}
I20260929 20:27:32.325017 4203 heartbeater.cc:347] Connected to a master server at 127.17.53.126:39255
I20260929 20:27:32.325322 4203 heartbeater.cc:464] Registering TS with master...
I20260929 20:27:32.326050 4203 heartbeater.cc:511] Master 127.17.53.126:39255 requested a full tablet report, sending...
I20260929 20:27:32.328681 4016 ts_manager.cc:194] Registered new tserver with Master: 2eb6300c78c4445e9c3d409a041c1a56 (127.17.53.65:45935)
I20260929 20:27:32.331069 4016 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.17.53.65:37817
W20260929 20:27:32.646253 4207 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:27:32.646534 4207 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:27:32.646620 4207 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:27:32.653965 4207 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:27:32.654062 4207 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:27:32.654248 4207 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.53.66
I20260929 20:27:32.663411 4207 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.53.84:34029
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790713339161492-17621-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790713339161492-17621-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.53.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790713339161492-17621-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.17.53.66
--webserver_port=0
--tserver_master_addrs=127.17.53.126:39255
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.53.66
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790713339161492-17621-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:27:32.664675 4207 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:27:32.666432 4207 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:27:32.680143 4213 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:27:32.680814 4212 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:27:32.681288 4215 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:27:32.682283 4207 server_base.cc:1034] running on GCE node
I20260929 20:27:32.683313 4207 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:27:32.684801 4207 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:27:32.686403 4207 hybrid_clock.cc:648] HybridClock initialized: now 1790713652686233 us; error 136 us; skew 500 ppm
I20260929 20:27:32.686779 4207 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 20:27:32.689931 4207 webserver.cc:460] Webserver started at http://127.17.53.66:41729/ using document root <none> and password file <none>
I20260929 20:27:32.691100 4207 fs_manager.cc:359] Metadata directory not provided
I20260929 20:27:32.691264 4207 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:27:32.691891 4207 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:27:32.696986 4207 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790713339161492-17621-0/minicluster-data/ts-1/data/instance:
uuid: "0feca1020f4b4426974a72e85aa6975d"
format_stamp: "Formatted at 2026-09-29 20:27:32 on dist-test-slave-zbnc"
server_key: "f3ecc6d1e9ff1f491825090af8e39ab8"
server_key_iv: "af5ec934b515e9012f9f06c1e36f4210"
server_key_version: "encryptionkey@0"
I20260929 20:27:32.698145 4207 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790713339161492-17621-0/minicluster-data/ts-1/wal/instance:
uuid: "0feca1020f4b4426974a72e85aa6975d"
format_stamp: "Formatted at 2026-09-29 20:27:32 on dist-test-slave-zbnc"
server_key: "f3ecc6d1e9ff1f491825090af8e39ab8"
server_key_iv: "af5ec934b515e9012f9f06c1e36f4210"
server_key_version: "encryptionkey@0"
I20260929 20:27:32.705929 4207 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.003s
I20260929 20:27:32.711470 4221 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:27:32.713318 4207 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260929 20:27:32.713519 4207 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790713339161492-17621-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790713339161492-17621-0/minicluster-data/ts-1/wal
uuid: "0feca1020f4b4426974a72e85aa6975d"
format_stamp: "Formatted at 2026-09-29 20:27:32 on dist-test-slave-zbnc"
server_key: "f3ecc6d1e9ff1f491825090af8e39ab8"
server_key_iv: "af5ec934b515e9012f9f06c1e36f4210"
server_key_version: "encryptionkey@0"
I20260929 20:27:32.713806 4207 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790713339161492-17621-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790713339161492-17621-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790713339161492-17621-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:27:32.737129 4207 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:27:32.738140 4207 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:27:32.738487 4207 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:27:32.739689 4207 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:27:32.741312 4207 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:27:32.741428 4207 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:27:32.741542 4207 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:27:32.741621 4207 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:27:32.776588 4207 rpc_server.cc:307] RPC server started. Bound to: 127.17.53.66:37193
I20260929 20:27:32.776631 4333 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.53.66:37193 every 8 connection(s)
I20260929 20:27:32.778052 4207 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790713339161492-17621-0/minicluster-data/ts-1/data/info.pb
I20260929 20:27:32.782902 17621 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu as pid 4207
I20260929 20:27:32.783106 17621 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790713339161492-17621-0/minicluster-data/ts-1/wal/instance
I20260929 20:27:32.783753 17621 external_mini_cluster.cc:1428] Setting key d9c6ecfbc3d53563320f2320d2c9b092
I20260929 20:27:32.787096 17621 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
/tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790713339161492-17621-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790713339161492-17621-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790713339161492-17621-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790713339161492-17621-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.53.67:0
--local_ip_for_outbound_sockets=127.17.53.67
--webserver_interface=127.17.53.67
--webserver_port=0
--tserver_master_addrs=127.17.53.126:39255
--builtin_ntp_servers=127.17.53.84:34029
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 20:27:32.793422 4205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:27:32.780888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.53.66:59193 (local address 127.17.53.126:39255)
0929 20:27:32.781077 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0929 20:27:32.781081 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:27:32.782732 (+ 1651us) server_negotiation.cc:407] Connection header received
0929 20:27:32.785294 (+ 2562us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:27:32.785299 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:27:32.785368 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:27:32.785468 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:27:32.787024 (+ 1556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:27:32.787795 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:27:32.790493 (+ 2698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:27:32.790930 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:27:32.791042 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:27:32.792679 (+ 1637us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:27:32.792683 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:27:32.792686 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:27:32.792704 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:27:32.792734 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:27:32.792745 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:27:32.792814 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0929 20:27:32.793122 (+ 308us) server_negotiation.cc:299] Negotiation successful
0929 20:27:32.793198 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260929 20:27:32.795079 4336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:27:32.781541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:39255 (local address 127.17.53.66:59193)
0929 20:27:32.782533 (+ 992us) negotiation.cc:107] Waiting for socket to connect
0929 20:27:32.782595 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0929 20:27:32.785030 (+ 2435us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:27:32.785747 (+ 717us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:27:32.785756 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:27:32.785938 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:27:32.786795 (+ 857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:27:32.786825 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:27:32.787961 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:27:32.787970 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:27:32.790351 (+ 2381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:27:32.790369 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:27:32.791152 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:27:32.791161 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:27:32.791509 (+ 348us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:27:32.792301 (+ 792us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 20:27:32.792308 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:27:32.792366 (+ 58us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 20:27:32.792476 (+ 110us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:27:32.792840 (+ 364us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:27:32.792862 (+ 22us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:27:32.792873 (+ 11us) client_negotiation.cc:769] Sending connection context
0929 20:27:32.793118 (+ 245us) client_negotiation.cc:240] Negotiation successful
0929 20:27:32.793561 (+ 443us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":474,"thread_start_us":298,"threads_started":1}
I20260929 20:27:32.796622 4334 heartbeater.cc:347] Connected to a master server at 127.17.53.126:39255
I20260929 20:27:32.796931 4334 heartbeater.cc:464] Registering TS with master...
I20260929 20:27:32.797705 4334 heartbeater.cc:511] Master 127.17.53.126:39255 requested a full tablet report, sending...
I20260929 20:27:32.799234 4016 ts_manager.cc:194] Registered new tserver with Master: 0feca1020f4b4426974a72e85aa6975d (127.17.53.66:37193)
I20260929 20:27:32.800041 4016 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.17.53.66:59193
W20260929 20:27:33.118037 4337 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:27:33.118397 4337 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:27:33.118575 4337 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:27:33.126904 4337 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:27:33.127067 4337 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:27:33.127247 4337 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.53.67
I20260929 20:27:33.137395 4337 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.53.84:34029
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790713339161492-17621-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790713339161492-17621-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.53.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790713339161492-17621-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.17.53.67
--webserver_port=0
--tserver_master_addrs=127.17.53.126:39255
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.53.67
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790713339161492-17621-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:27:33.138794 4337 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:27:33.140681 4337 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:27:33.154330 4342 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:27:33.154273 4345 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:27:33.154764 4343 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:27:33.155169 4337 server_base.cc:1034] running on GCE node
I20260929 20:27:33.155890 4337 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:27:33.156934 4337 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:27:33.158191 4337 hybrid_clock.cc:648] HybridClock initialized: now 1790713653158148 us; error 47 us; skew 500 ppm
I20260929 20:27:33.158607 4337 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 20:27:33.160914 4337 webserver.cc:460] Webserver started at http://127.17.53.67:41501/ using document root <none> and password file <none>
I20260929 20:27:33.161599 4337 fs_manager.cc:359] Metadata directory not provided
I20260929 20:27:33.161718 4337 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:27:33.162030 4337 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:27:33.164969 4337 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790713339161492-17621-0/minicluster-data/ts-2/data/instance:
uuid: "b2339d533f554ac98c7cb0df03363ae6"
format_stamp: "Formatted at 2026-09-29 20:27:33 on dist-test-slave-zbnc"
server_key: "912586db709eae325296ff655a74498f"
server_key_iv: "c9c87922e1e7e7e3a62e07d0eb56b6f1"
server_key_version: "encryptionkey@0"
I20260929 20:27:33.165750 4337 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790713339161492-17621-0/minicluster-data/ts-2/wal/instance:
uuid: "b2339d533f554ac98c7cb0df03363ae6"
format_stamp: "Formatted at 2026-09-29 20:27:33 on dist-test-slave-zbnc"
server_key: "912586db709eae325296ff655a74498f"
server_key_iv: "c9c87922e1e7e7e3a62e07d0eb56b6f1"
server_key_version: "encryptionkey@0"
I20260929 20:27:33.170944 4337 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.004s
I20260929 20:27:33.174228 4351 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:27:33.175619 4337 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 20:27:33.175786 4337 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790713339161492-17621-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790713339161492-17621-0/minicluster-data/ts-2/wal
uuid: "b2339d533f554ac98c7cb0df03363ae6"
format_stamp: "Formatted at 2026-09-29 20:27:33 on dist-test-slave-zbnc"
server_key: "912586db709eae325296ff655a74498f"
server_key_iv: "c9c87922e1e7e7e3a62e07d0eb56b6f1"
server_key_version: "encryptionkey@0"
I20260929 20:27:33.176009 4337 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790713339161492-17621-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790713339161492-17621-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790713339161492-17621-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:27:33.200577 4337 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:27:33.201418 4337 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:27:33.201721 4337 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:27:33.202675 4337 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:27:33.204113 4337 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:27:33.204233 4337 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:27:33.204382 4337 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:27:33.204479 4337 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:27:33.237305 4337 rpc_server.cc:307] RPC server started. Bound to: 127.17.53.67:43643
I20260929 20:27:33.237371 4463 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.53.67:43643 every 8 connection(s)
I20260929 20:27:33.238653 4337 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790713339161492-17621-0/minicluster-data/ts-2/data/info.pb
I20260929 20:27:33.244175 17621 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu as pid 4337
I20260929 20:27:33.244359 17621 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790713339161492-17621-0/minicluster-data/ts-2/wal/instance
I20260929 20:27:33.244779 17621 external_mini_cluster.cc:1428] Setting key bb0facf15ab4841878bcd54f705e63a5
I20260929 20:27:33.251575 4205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:27:33.241210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.53.67:46949 (local address 127.17.53.126:39255)
0929 20:27:33.241418 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0929 20:27:33.241424 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:27:33.243057 (+ 1633us) server_negotiation.cc:407] Connection header received
0929 20:27:33.245154 (+ 2097us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:27:33.245160 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:27:33.245249 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:27:33.245338 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:27:33.246776 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:27:33.247711 (+ 935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:27:33.249181 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:27:33.249587 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:27:33.249673 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:27:33.250938 (+ 1265us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:27:33.250943 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:27:33.250946 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:27:33.250963 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:27:33.251001 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:27:33.251013 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:27:33.251082 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0929 20:27:33.251271 (+ 189us) server_negotiation.cc:299] Negotiation successful
0929 20:27:33.251344 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 20:27:33.252655 4466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:27:33.241690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:39255 (local address 127.17.53.67:46949)
0929 20:27:33.242752 (+ 1062us) negotiation.cc:107] Waiting for socket to connect
0929 20:27:33.242915 (+ 163us) client_negotiation.cc:174] Beginning negotiation
0929 20:27:33.244844 (+ 1929us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:27:33.245611 (+ 767us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:27:33.245619 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:27:33.245744 (+ 125us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:27:33.246591 (+ 847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:27:33.246617 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:27:33.247837 (+ 1220us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:27:33.247841 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:27:33.249056 (+ 1215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:27:33.249073 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:27:33.249757 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:27:33.249762 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:27:33.250000 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:27:33.250626 (+ 626us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 20:27:33.250628 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:27:33.250664 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 20:27:33.250740 (+ 76us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:27:33.251096 (+ 356us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:27:33.251104 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:27:33.251109 (+ 5us) client_negotiation.cc:769] Sending connection context
0929 20:27:33.251248 (+ 139us) client_negotiation.cc:240] Negotiation successful
0929 20:27:33.251523 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":494,"thread_start_us":289,"threads_started":1}
I20260929 20:27:33.254071 4464 heartbeater.cc:347] Connected to a master server at 127.17.53.126:39255
I20260929 20:27:33.254377 4464 heartbeater.cc:464] Registering TS with master...
I20260929 20:27:33.255221 4464 heartbeater.cc:511] Master 127.17.53.126:39255 requested a full tablet report, sending...
I20260929 20:27:33.256618 4016 ts_manager.cc:194] Registered new tserver with Master: b2339d533f554ac98c7cb0df03363ae6 (127.17.53.67:43643)
I20260929 20:27:33.257423 4016 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.17.53.67:46949
I20260929 20:27:33.259790 17621 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 20:27:33.269328 4205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:27:33.263625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52528 (local address 127.17.53.126:39255)
0929 20:27:33.263833 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0929 20:27:33.263842 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 20:27:33.264204 (+ 362us) server_negotiation.cc:407] Connection header received
0929 20:27:33.264340 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:27:33.264344 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:27:33.264402 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:27:33.264473 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:27:33.264946 (+ 473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:27:33.265628 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:27:33.267101 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:27:33.267483 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:27:33.267578 (+ 95us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:27:33.268606 (+ 1028us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:27:33.268611 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:27:33.268614 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:27:33.268632 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:27:33.268663 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:27:33.268674 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:27:33.268783 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0929 20:27:33.268959 (+ 176us) server_negotiation.cc:299] Negotiation successful
0929 20:27:33.269032 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 20:27:33.272974 4016 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52528:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 20:27:33.279923 4016 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 20:27:33.319433 4476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:27:33.291832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.65:45935 (local address 127.0.0.1:60844)
0929 20:27:33.307067 (+ 15235us) negotiation.cc:107] Waiting for socket to connect
0929 20:27:33.307119 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0929 20:27:33.307339 (+ 220us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:27:33.309912 (+ 2573us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:27:33.309921 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:27:33.309955 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:27:33.310429 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:27:33.310444 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:27:33.312812 (+ 2368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:27:33.312817 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:27:33.315768 (+ 2951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:27:33.315791 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:27:33.318401 (+ 2610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:27:33.318408 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:27:33.318624 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:27:33.318687 (+ 63us) client_negotiation.cc:769] Sending connection context
0929 20:27:33.318818 (+ 131us) client_negotiation.cc:240] Negotiation successful
0929 20:27:33.319028 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14862,"mutex_wait_us":120,"thread_start_us":15719,"threads_started":1}
I20260929 20:27:33.320116 4478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:27:33.292023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60844 (local address 127.17.53.65:45935)
0929 20:27:33.309231 (+ 17208us) server_negotiation.cc:206] Beginning negotiation
0929 20:27:33.309241 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 20:27:33.309276 (+ 35us) server_negotiation.cc:407] Connection header received
0929 20:27:33.309389 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:27:33.309398 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:27:33.309672 (+ 274us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:27:33.310646 (+ 974us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:27:33.310929 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:27:33.312661 (+ 1732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:27:33.316081 (+ 3420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:27:33.318126 (+ 2045us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:27:33.318991 (+ 865us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:27:33.319267 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0929 20:27:33.319338 (+ 71us) server_negotiation.cc:299] Negotiation successful
0929 20:27:33.319763 (+ 425us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17031,"thread_start_us":16916,"threads_started":1}
I20260929 20:27:33.321741 4479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:27:33.292658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46390 (local address 127.17.53.67:43643)
0929 20:27:33.308759 (+ 16101us) server_negotiation.cc:206] Beginning negotiation
0929 20:27:33.308772 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 20:27:33.309766 (+ 994us) server_negotiation.cc:407] Connection header received
0929 20:27:33.309982 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:27:33.309991 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:27:33.310226 (+ 235us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:27:33.310428 (+ 202us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:27:33.311428 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:27:33.313021 (+ 1593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:27:33.318948 (+ 5927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:27:33.320409 (+ 1461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:27:33.320541 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:27:33.320724 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0929 20:27:33.321251 (+ 527us) server_negotiation.cc:299] Negotiation successful
0929 20:27:33.321417 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15897,"thread_start_us":15744,"threads_started":1}
I20260929 20:27:33.322826 4477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:27:33.292036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.67:43643 (local address 127.0.0.1:46390)
0929 20:27:33.309483 (+ 17447us) negotiation.cc:107] Waiting for socket to connect
0929 20:27:33.309513 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0929 20:27:33.309631 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:27:33.310731 (+ 1100us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:27:33.310737 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:27:33.310761 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:27:33.311212 (+ 451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:27:33.311228 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:27:33.313174 (+ 1946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:27:33.313185 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:27:33.318666 (+ 5481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:27:33.318678 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:27:33.320869 (+ 2191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:27:33.320874 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:27:33.321048 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:27:33.321071 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 20:27:33.321162 (+ 91us) client_negotiation.cc:240] Negotiation successful
0929 20:27:33.321293 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17340,"mutex_wait_us":3040,"thread_start_us":22834,"threads_started":1}
I20260929 20:27:33.324566 4138 tablet_service.cc:1467] Processing CreateTablet for tablet e09a5f59ed39486fab4363896525cf62 (DEFAULT_TABLE table=test-table [id=5f96ad4d38304764897a4830b09b4949]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:27:33.326144 4138 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e09a5f59ed39486fab4363896525cf62. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:27:33.326004 4399 tablet_service.cc:1467] Processing CreateTablet for tablet e09a5f59ed39486fab4363896525cf62 (DEFAULT_TABLE table=test-table [id=5f96ad4d38304764897a4830b09b4949]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:27:33.326611 4480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:27:33.293772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.66:37193 (local address 127.0.0.1:34276)
0929 20:27:33.318520 (+ 24748us) negotiation.cc:107] Waiting for socket to connect
0929 20:27:33.318545 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0929 20:27:33.318678 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:27:33.319534 (+ 856us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:27:33.319541 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:27:33.319570 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:27:33.320048 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:27:33.320063 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:27:33.322052 (+ 1989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:27:33.322056 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:27:33.323685 (+ 1629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:27:33.323698 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:27:33.325894 (+ 2196us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:27:33.325900 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:27:33.326067 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:27:33.326103 (+ 36us) client_negotiation.cc:769] Sending connection context
0929 20:27:33.326207 (+ 104us) client_negotiation.cc:240] Negotiation successful
0929 20:27:33.326328 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24614,"thread_start_us":24435,"threads_started":1}
I20260929 20:27:33.327162 4481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:27:33.294511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34276 (local address 127.17.53.66:37193)
0929 20:27:33.318110 (+ 23599us) server_negotiation.cc:206] Beginning negotiation
0929 20:27:33.318123 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 20:27:33.318669 (+ 546us) server_negotiation.cc:407] Connection header received
0929 20:27:33.318850 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:27:33.318882 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:27:33.319115 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:27:33.319306 (+ 191us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:27:33.320223 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:27:33.321811 (+ 1588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:27:33.324026 (+ 2215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:27:33.325380 (+ 1354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:27:33.325499 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:27:33.325721 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0929 20:27:33.326356 (+ 635us) server_negotiation.cc:299] Negotiation successful
0929 20:27:33.326620 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23448,"thread_start_us":23328,"threads_started":1}
I20260929 20:27:33.327390 4399 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e09a5f59ed39486fab4363896525cf62. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:27:33.330513 4269 tablet_service.cc:1467] Processing CreateTablet for tablet e09a5f59ed39486fab4363896525cf62 (DEFAULT_TABLE table=test-table [id=5f96ad4d38304764897a4830b09b4949]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:27:33.331830 4269 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e09a5f59ed39486fab4363896525cf62. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:27:33.334175 4203 heartbeater.cc:503] Master 127.17.53.126:39255 was elected leader, sending a full tablet report...
I20260929 20:27:33.338956 4482 tablet_bootstrap.cc:492] T e09a5f59ed39486fab4363896525cf62 P b2339d533f554ac98c7cb0df03363ae6: Bootstrap starting.
I20260929 20:27:33.340945 4483 tablet_bootstrap.cc:492] T e09a5f59ed39486fab4363896525cf62 P 2eb6300c78c4445e9c3d409a041c1a56: Bootstrap starting.
I20260929 20:27:33.343883 4482 tablet_bootstrap.cc:654] T e09a5f59ed39486fab4363896525cf62 P b2339d533f554ac98c7cb0df03363ae6: Neither blocks nor log segments found. Creating new log.
I20260929 20:27:33.344498 4483 tablet_bootstrap.cc:654] T e09a5f59ed39486fab4363896525cf62 P 2eb6300c78c4445e9c3d409a041c1a56: Neither blocks nor log segments found. Creating new log.
I20260929 20:27:33.345201 4482 log.cc:824] T e09a5f59ed39486fab4363896525cf62 P b2339d533f554ac98c7cb0df03363ae6: Log is configured to *not* fsync() on all Append() calls
I20260929 20:27:33.345676 4483 log.cc:824] T e09a5f59ed39486fab4363896525cf62 P 2eb6300c78c4445e9c3d409a041c1a56: Log is configured to *not* fsync() on all Append() calls
I20260929 20:27:33.348152 4484 tablet_bootstrap.cc:492] T e09a5f59ed39486fab4363896525cf62 P 0feca1020f4b4426974a72e85aa6975d: Bootstrap starting.
I20260929 20:27:33.349637 4483 tablet_bootstrap.cc:492] T e09a5f59ed39486fab4363896525cf62 P 2eb6300c78c4445e9c3d409a041c1a56: No bootstrap required, opened a new log
I20260929 20:27:33.349910 4483 ts_tablet_manager.cc:1397] T e09a5f59ed39486fab4363896525cf62 P 2eb6300c78c4445e9c3d409a041c1a56: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.004s
I20260929 20:27:33.350597 4482 tablet_bootstrap.cc:492] T e09a5f59ed39486fab4363896525cf62 P b2339d533f554ac98c7cb0df03363ae6: No bootstrap required, opened a new log
I20260929 20:27:33.350888 4482 ts_tablet_manager.cc:1397] T e09a5f59ed39486fab4363896525cf62 P b2339d533f554ac98c7cb0df03363ae6: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.005s
I20260929 20:27:33.352308 4484 tablet_bootstrap.cc:654] T e09a5f59ed39486fab4363896525cf62 P 0feca1020f4b4426974a72e85aa6975d: Neither blocks nor log segments found. Creating new log.
I20260929 20:27:33.353564 4484 log.cc:824] T e09a5f59ed39486fab4363896525cf62 P 0feca1020f4b4426974a72e85aa6975d: Log is configured to *not* fsync() on all Append() calls
I20260929 20:27:33.357517 4484 tablet_bootstrap.cc:492] T e09a5f59ed39486fab4363896525cf62 P 0feca1020f4b4426974a72e85aa6975d: No bootstrap required, opened a new log
I20260929 20:27:33.357810 4484 ts_tablet_manager.cc:1397] T e09a5f59ed39486fab4363896525cf62 P 0feca1020f4b4426974a72e85aa6975d: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260929 20:27:33.358362 4483 raft_consensus.cc:348] T e09a5f59ed39486fab4363896525cf62 P 2eb6300c78c4445e9c3d409a041c1a56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2eb6300c78c4445e9c3d409a041c1a56" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 45935 } } peers { permanent_uuid: "b2339d533f554ac98c7cb0df03363ae6" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 43643 } } peers { permanent_uuid: "0feca1020f4b4426974a72e85aa6975d" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 37193 } }
I20260929 20:27:33.358768 4483 raft_consensus.cc:374] T e09a5f59ed39486fab4363896525cf62 P 2eb6300c78c4445e9c3d409a041c1a56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:27:33.358887 4483 raft_consensus.cc:729] T e09a5f59ed39486fab4363896525cf62 P 2eb6300c78c4445e9c3d409a041c1a56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2eb6300c78c4445e9c3d409a041c1a56, State: Initialized, Role: FOLLOWER
I20260929 20:27:33.359556 4483 consensus_queue.cc:260] T e09a5f59ed39486fab4363896525cf62 P 2eb6300c78c4445e9c3d409a041c1a56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2eb6300c78c4445e9c3d409a041c1a56" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 45935 } } peers { permanent_uuid: "b2339d533f554ac98c7cb0df03363ae6" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 43643 } } peers { permanent_uuid: "0feca1020f4b4426974a72e85aa6975d" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 37193 } }
I20260929 20:27:33.361478 4482 raft_consensus.cc:348] T e09a5f59ed39486fab4363896525cf62 P b2339d533f554ac98c7cb0df03363ae6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2eb6300c78c4445e9c3d409a041c1a56" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 45935 } } peers { permanent_uuid: "b2339d533f554ac98c7cb0df03363ae6" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 43643 } } peers { permanent_uuid: "0feca1020f4b4426974a72e85aa6975d" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 37193 } }
I20260929 20:27:33.361922 4482 raft_consensus.cc:374] T e09a5f59ed39486fab4363896525cf62 P b2339d533f554ac98c7cb0df03363ae6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:27:33.362033 4482 raft_consensus.cc:729] T e09a5f59ed39486fab4363896525cf62 P b2339d533f554ac98c7cb0df03363ae6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2339d533f554ac98c7cb0df03363ae6, State: Initialized, Role: FOLLOWER
I20260929 20:27:33.362620 4482 consensus_queue.cc:260] T e09a5f59ed39486fab4363896525cf62 P b2339d533f554ac98c7cb0df03363ae6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2eb6300c78c4445e9c3d409a041c1a56" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 45935 } } peers { permanent_uuid: "b2339d533f554ac98c7cb0df03363ae6" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 43643 } } peers { permanent_uuid: "0feca1020f4b4426974a72e85aa6975d" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 37193 } }
I20260929 20:27:33.363893 4464 heartbeater.cc:503] Master 127.17.53.126:39255 was elected leader, sending a full tablet report...
I20260929 20:27:33.364517 4482 ts_tablet_manager.cc:1428] T e09a5f59ed39486fab4363896525cf62 P b2339d533f554ac98c7cb0df03363ae6: Time spent starting tablet: real 0.014s user 0.004s sys 0.009s
I20260929 20:27:33.366554 4483 ts_tablet_manager.cc:1428] T e09a5f59ed39486fab4363896525cf62 P 2eb6300c78c4445e9c3d409a041c1a56: Time spent starting tablet: real 0.017s user 0.013s sys 0.004s
I20260929 20:27:33.368953 4484 raft_consensus.cc:348] T e09a5f59ed39486fab4363896525cf62 P 0feca1020f4b4426974a72e85aa6975d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2eb6300c78c4445e9c3d409a041c1a56" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 45935 } } peers { permanent_uuid: "b2339d533f554ac98c7cb0df03363ae6" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 43643 } } peers { permanent_uuid: "0feca1020f4b4426974a72e85aa6975d" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 37193 } }
I20260929 20:27:33.369436 4484 raft_consensus.cc:374] T e09a5f59ed39486fab4363896525cf62 P 0feca1020f4b4426974a72e85aa6975d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:27:33.369560 4484 raft_consensus.cc:729] T e09a5f59ed39486fab4363896525cf62 P 0feca1020f4b4426974a72e85aa6975d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0feca1020f4b4426974a72e85aa6975d, State: Initialized, Role: FOLLOWER
I20260929 20:27:33.370239 4484 consensus_queue.cc:260] T e09a5f59ed39486fab4363896525cf62 P 0feca1020f4b4426974a72e85aa6975d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2eb6300c78c4445e9c3d409a041c1a56" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 45935 } } peers { permanent_uuid: "b2339d533f554ac98c7cb0df03363ae6" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 43643 } } peers { permanent_uuid: "0feca1020f4b4426974a72e85aa6975d" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 37193 } }
I20260929 20:27:33.371690 4334 heartbeater.cc:503] Master 127.17.53.126:39255 was elected leader, sending a full tablet report...
I20260929 20:27:33.372396 4484 ts_tablet_manager.cc:1428] T e09a5f59ed39486fab4363896525cf62 P 0feca1020f4b4426974a72e85aa6975d: Time spent starting tablet: real 0.014s user 0.014s sys 0.000s
I20260929 20:27:33.384755 4489 raft_consensus.cc:482] T e09a5f59ed39486fab4363896525cf62 P b2339d533f554ac98c7cb0df03363ae6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 20:27:33.385056 4489 raft_consensus.cc:504] T e09a5f59ed39486fab4363896525cf62 P b2339d533f554ac98c7cb0df03363ae6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2eb6300c78c4445e9c3d409a041c1a56" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 45935 } } peers { permanent_uuid: "b2339d533f554ac98c7cb0df03363ae6" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 43643 } } peers { permanent_uuid: "0feca1020f4b4426974a72e85aa6975d" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 37193 } }
I20260929 20:27:33.386731 4489 leader_election.cc:288] T e09a5f59ed39486fab4363896525cf62 P b2339d533f554ac98c7cb0df03363ae6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2eb6300c78c4445e9c3d409a041c1a56 (127.17.53.65:45935), 0feca1020f4b4426974a72e85aa6975d (127.17.53.66:37193)
I20260929 20:27:33.392813 4466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:27:33.387021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.65:45935 (local address 127.17.53.67:54897)
0929 20:27:33.387299 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0929 20:27:33.387320 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 20:27:33.387467 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:27:33.387884 (+ 417us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:27:33.387888 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:27:33.387911 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:27:33.388145 (+ 234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:27:33.388159 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:27:33.389369 (+ 1210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:27:33.389374 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:27:33.391038 (+ 1664us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:27:33.391050 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:27:33.392246 (+ 1196us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:27:33.392250 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:27:33.392380 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:27:33.392405 (+ 25us) client_negotiation.cc:769] Sending connection context
0929 20:27:33.392473 (+ 68us) client_negotiation.cc:240] Negotiation successful
0929 20:27:33.392595 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"mutex_wait_us":23,"spinlock_wait_cycles":6016}
I20260929 20:27:33.392899 4478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:27:33.387285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.53.67:54897 (local address 127.17.53.65:45935)
0929 20:27:33.387485 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0929 20:27:33.387490 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:27:33.387513 (+ 23us) server_negotiation.cc:407] Connection header received
0929 20:27:33.387598 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:27:33.387605 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:27:33.387702 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:27:33.387811 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:27:33.388328 (+ 517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:27:33.389244 (+ 916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:27:33.391187 (+ 1943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:27:33.392112 (+ 925us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:27:33.392281 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:27:33.392366 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0929 20:27:33.392534 (+ 168us) server_negotiation.cc:299] Negotiation successful
0929 20:27:33.392665 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260929 20:27:33.393965 4492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:27:33.387079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.66:37193 (local address 127.17.53.67:51345)
0929 20:27:33.387856 (+ 777us) negotiation.cc:107] Waiting for socket to connect
0929 20:27:33.387876 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 20:27:33.388011 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:27:33.388340 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:27:33.388344 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:27:33.388369 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:27:33.388747 (+ 378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:27:33.388758 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:27:33.389871 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:27:33.389875 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:27:33.391957 (+ 2082us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:27:33.391969 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:27:33.393436 (+ 1467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:27:33.393443 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:27:33.393581 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:27:33.393604 (+ 23us) client_negotiation.cc:769] Sending connection context
0929 20:27:33.393675 (+ 71us) client_negotiation.cc:240] Negotiation successful
0929 20:27:33.393761 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":645,"thread_start_us":444,"threads_started":1}
I20260929 20:27:33.394112 4481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:27:33.387152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.53.67:51345 (local address 127.17.53.66:37193)
0929 20:27:33.387337 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0929 20:27:33.387344 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:27:33.387965 (+ 621us) server_negotiation.cc:407] Connection header received
0929 20:27:33.388135 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:27:33.388140 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:27:33.388229 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:27:33.388317 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:27:33.388888 (+ 571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:27:33.389723 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:27:33.392137 (+ 2414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:27:33.393280 (+ 1143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:27:33.393413 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:27:33.393489 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0929 20:27:33.393729 (+ 240us) server_negotiation.cc:299] Negotiation successful
0929 20:27:33.393870 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 20:27:33.394006 4158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e09a5f59ed39486fab4363896525cf62" candidate_uuid: "b2339d533f554ac98c7cb0df03363ae6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2eb6300c78c4445e9c3d409a041c1a56" is_pre_election: true
I20260929 20:27:33.394572 4158 raft_consensus.cc:2454] T e09a5f59ed39486fab4363896525cf62 P 2eb6300c78c4445e9c3d409a041c1a56 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b2339d533f554ac98c7cb0df03363ae6 in term 0.
I20260929 20:27:33.395165 4289 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e09a5f59ed39486fab4363896525cf62" candidate_uuid: "b2339d533f554ac98c7cb0df03363ae6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0feca1020f4b4426974a72e85aa6975d" is_pre_election: true
I20260929 20:27:33.395315 4355 leader_election.cc:302] T e09a5f59ed39486fab4363896525cf62 P b2339d533f554ac98c7cb0df03363ae6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2eb6300c78c4445e9c3d409a041c1a56, b2339d533f554ac98c7cb0df03363ae6; no voters:
I20260929 20:27:33.395642 4289 raft_consensus.cc:2454] T e09a5f59ed39486fab4363896525cf62 P 0feca1020f4b4426974a72e85aa6975d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b2339d533f554ac98c7cb0df03363ae6 in term 0.
I20260929 20:27:33.395764 4489 raft_consensus.cc:2781] T e09a5f59ed39486fab4363896525cf62 P b2339d533f554ac98c7cb0df03363ae6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 20:27:33.395902 4489 raft_consensus.cc:482] T e09a5f59ed39486fab4363896525cf62 P b2339d533f554ac98c7cb0df03363ae6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 20:27:33.396018 4489 raft_consensus.cc:3037] T e09a5f59ed39486fab4363896525cf62 P b2339d533f554ac98c7cb0df03363ae6 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:27:33.398234 4489 raft_consensus.cc:504] T e09a5f59ed39486fab4363896525cf62 P b2339d533f554ac98c7cb0df03363ae6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2eb6300c78c4445e9c3d409a041c1a56" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 45935 } } peers { permanent_uuid: "b2339d533f554ac98c7cb0df03363ae6" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 43643 } } peers { permanent_uuid: "0feca1020f4b4426974a72e85aa6975d" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 37193 } }
I20260929 20:27:33.398802 4489 leader_election.cc:288] T e09a5f59ed39486fab4363896525cf62 P b2339d533f554ac98c7cb0df03363ae6 [CANDIDATE]: Term 1 election: Requested vote from peers 2eb6300c78c4445e9c3d409a041c1a56 (127.17.53.65:45935), 0feca1020f4b4426974a72e85aa6975d (127.17.53.66:37193)
I20260929 20:27:33.399227 4158 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e09a5f59ed39486fab4363896525cf62" candidate_uuid: "b2339d533f554ac98c7cb0df03363ae6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2eb6300c78c4445e9c3d409a041c1a56"
I20260929 20:27:33.399276 4289 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e09a5f59ed39486fab4363896525cf62" candidate_uuid: "b2339d533f554ac98c7cb0df03363ae6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0feca1020f4b4426974a72e85aa6975d"
I20260929 20:27:33.399418 4158 raft_consensus.cc:3037] T e09a5f59ed39486fab4363896525cf62 P 2eb6300c78c4445e9c3d409a041c1a56 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:27:33.399509 4289 raft_consensus.cc:3037] T e09a5f59ed39486fab4363896525cf62 P 0feca1020f4b4426974a72e85aa6975d [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:27:33.401669 4158 raft_consensus.cc:2454] T e09a5f59ed39486fab4363896525cf62 P 2eb6300c78c4445e9c3d409a041c1a56 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b2339d533f554ac98c7cb0df03363ae6 in term 1.
I20260929 20:27:33.401777 4289 raft_consensus.cc:2454] T e09a5f59ed39486fab4363896525cf62 P 0feca1020f4b4426974a72e85aa6975d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b2339d533f554ac98c7cb0df03363ae6 in term 1.
I20260929 20:27:33.402060 4355 leader_election.cc:302] T e09a5f59ed39486fab4363896525cf62 P b2339d533f554ac98c7cb0df03363ae6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2eb6300c78c4445e9c3d409a041c1a56, b2339d533f554ac98c7cb0df03363ae6; no voters:
I20260929 20:27:33.402369 4489 raft_consensus.cc:2781] T e09a5f59ed39486fab4363896525cf62 P b2339d533f554ac98c7cb0df03363ae6 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:27:33.402788 4489 raft_consensus.cc:686] T e09a5f59ed39486fab4363896525cf62 P b2339d533f554ac98c7cb0df03363ae6 [term 1 LEADER]: Becoming Leader. State: Replica: b2339d533f554ac98c7cb0df03363ae6, State: Running, Role: LEADER
I20260929 20:27:33.403358 4489 consensus_queue.cc:237] T e09a5f59ed39486fab4363896525cf62 P b2339d533f554ac98c7cb0df03363ae6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2eb6300c78c4445e9c3d409a041c1a56" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 45935 } } peers { permanent_uuid: "b2339d533f554ac98c7cb0df03363ae6" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 43643 } } peers { permanent_uuid: "0feca1020f4b4426974a72e85aa6975d" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 37193 } }
I20260929 20:27:33.407956 4016 catalog_manager.cc:5495] T e09a5f59ed39486fab4363896525cf62 P b2339d533f554ac98c7cb0df03363ae6 reported cstate change: term changed from 0 to 1, leader changed from <none> to b2339d533f554ac98c7cb0df03363ae6 (127.17.53.67). New cstate: current_term: 1 leader_uuid: "b2339d533f554ac98c7cb0df03363ae6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2eb6300c78c4445e9c3d409a041c1a56" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 45935 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b2339d533f554ac98c7cb0df03363ae6" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 43643 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0feca1020f4b4426974a72e85aa6975d" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 37193 } health_report { overall_health: UNKNOWN } } }
I20260929 20:27:33.427945 4479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:27:33.422281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46402 (local address 127.17.53.67:43643)
0929 20:27:33.422477 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0929 20:27:33.422503 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0929 20:27:33.422577 (+ 74us) server_negotiation.cc:407] Connection header received
0929 20:27:33.422812 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:27:33.422817 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:27:33.422955 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:27:33.423104 (+ 149us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 20:27:33.423498 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:27:33.424251 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:27:33.425661 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:27:33.426102 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:27:33.426229 (+ 127us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:27:33.426815 (+ 586us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 20:27:33.427376 (+ 561us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 20:27:33.427568 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0929 20:27:33.427682 (+ 114us) server_negotiation.cc:299] Negotiation successful
0929 20:27:33.427725 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260929 20:27:33.436216 4289 raft_consensus.cc:1261] T e09a5f59ed39486fab4363896525cf62 P 0feca1020f4b4426974a72e85aa6975d [term 1 FOLLOWER]: Refusing update from remote peer b2339d533f554ac98c7cb0df03363ae6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:27:33.436299 4158 raft_consensus.cc:1261] T e09a5f59ed39486fab4363896525cf62 P 2eb6300c78c4445e9c3d409a041c1a56 [term 1 FOLLOWER]: Refusing update from remote peer b2339d533f554ac98c7cb0df03363ae6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:27:33.437148 4493 consensus_queue.cc:1035] T e09a5f59ed39486fab4363896525cf62 P b2339d533f554ac98c7cb0df03363ae6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2eb6300c78c4445e9c3d409a041c1a56" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 45935 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:27:33.437477 4489 consensus_queue.cc:1035] T e09a5f59ed39486fab4363896525cf62 P b2339d533f554ac98c7cb0df03363ae6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0feca1020f4b4426974a72e85aa6975d" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 37193 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:27:33.467705 4015 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:52528:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 20:27:33.468003 4015 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:52528:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 20:27:33.471712 4015 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b046fe18e38a456db218e661faed4a28: Sending DeleteTablet for 3 replicas of tablet e09a5f59ed39486fab4363896525cf62
I20260929 20:27:33.472868 4399 tablet_service.cc:1514] Processing DeleteTablet for tablet e09a5f59ed39486fab4363896525cf62 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:27:33 UTC) from {username='slave'} at 127.0.0.1:46390
I20260929 20:27:33.473166 4138 tablet_service.cc:1514] Processing DeleteTablet for tablet e09a5f59ed39486fab4363896525cf62 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:27:33 UTC) from {username='slave'} at 127.0.0.1:60844
I20260929 20:27:33.473321 4269 tablet_service.cc:1514] Processing DeleteTablet for tablet e09a5f59ed39486fab4363896525cf62 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:27:33 UTC) from {username='slave'} at 127.0.0.1:34276
I20260929 20:27:33.475327 17621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu pid 4075
I20260929 20:27:33.488379 4506 tablet_replica.cc:330] T e09a5f59ed39486fab4363896525cf62 P 2eb6300c78c4445e9c3d409a041c1a56: stopping tablet replica
I20260929 20:27:33.488884 4506 raft_consensus.cc:2229] T e09a5f59ed39486fab4363896525cf62 P 2eb6300c78c4445e9c3d409a041c1a56 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 20:27:33.489276 4507 tablet_replica.cc:330] T e09a5f59ed39486fab4363896525cf62 P 0feca1020f4b4426974a72e85aa6975d: stopping tablet replica
I20260929 20:27:33.489435 4506 raft_consensus.cc:2258] T e09a5f59ed39486fab4363896525cf62 P 2eb6300c78c4445e9c3d409a041c1a56 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:27:33.489689 4507 raft_consensus.cc:2229] T e09a5f59ed39486fab4363896525cf62 P 0feca1020f4b4426974a72e85aa6975d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 20:27:33.490154 4507 raft_consensus.cc:2258] T e09a5f59ed39486fab4363896525cf62 P 0feca1020f4b4426974a72e85aa6975d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:27:33.493547 4506 ts_tablet_manager.cc:1905] T e09a5f59ed39486fab4363896525cf62 P 2eb6300c78c4445e9c3d409a041c1a56: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:27:33.494220 4507 ts_tablet_manager.cc:1905] T e09a5f59ed39486fab4363896525cf62 P 0feca1020f4b4426974a72e85aa6975d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:27:33.494391 4478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:27:33.477133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60846 (local address 127.17.53.65:45935)
0929 20:27:33.477353 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0929 20:27:33.477362 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 20:27:33.481816 (+ 4454us) server_negotiation.cc:407] Connection header received
0929 20:27:33.482632 (+ 816us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:27:33.482641 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:27:33.482795 (+ 154us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:27:33.482906 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:27:33.484378 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:27:33.485991 (+ 1613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:27:33.489264 (+ 3273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:27:33.489930 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:27:33.490028 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:27:33.492447 (+ 2419us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:27:33.492461 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:27:33.492479 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:27:33.493324 (+ 845us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:27:33.493387 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:27:33.493412 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:27:33.493507 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0929 20:27:33.493863 (+ 356us) server_negotiation.cc:299] Negotiation successful
0929 20:27:33.493979 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260929 20:27:33.495829 4198 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:27:33.496119 4505 tablet_replica.cc:330] T e09a5f59ed39486fab4363896525cf62 P b2339d533f554ac98c7cb0df03363ae6: stopping tablet replica
I20260929 20:27:33.496557 4505 raft_consensus.cc:2229] T e09a5f59ed39486fab4363896525cf62 P b2339d533f554ac98c7cb0df03363ae6 [term 1 LEADER]: Raft consensus shutting down.
I20260929 20:27:33.497263 4505 raft_consensus.cc:2258] T e09a5f59ed39486fab4363896525cf62 P b2339d533f554ac98c7cb0df03363ae6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:27:33.499189 4505 ts_tablet_manager.cc:1905] T e09a5f59ed39486fab4363896525cf62 P b2339d533f554ac98c7cb0df03363ae6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:27:33.501549 4507 ts_tablet_manager.cc:1918] T e09a5f59ed39486fab4363896525cf62 P 0feca1020f4b4426974a72e85aa6975d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 20:27:33.501722 4507 log.cc:1196] T e09a5f59ed39486fab4363896525cf62 P 0feca1020f4b4426974a72e85aa6975d: Deleting WAL directory at /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790713339161492-17621-0/minicluster-data/ts-1/wal/wals/e09a5f59ed39486fab4363896525cf62
I20260929 20:27:33.502255 4507 ts_tablet_manager.cc:1939] T e09a5f59ed39486fab4363896525cf62 P 0feca1020f4b4426974a72e85aa6975d: Deleting consensus metadata
I20260929 20:27:33.504168 4001 catalog_manager.cc:4841] TS 0feca1020f4b4426974a72e85aa6975d (127.17.53.66:37193): tablet e09a5f59ed39486fab4363896525cf62 (table test-table [id=5f96ad4d38304764897a4830b09b4949]) successfully deleted
I20260929 20:27:33.507429 4505 ts_tablet_manager.cc:1918] T e09a5f59ed39486fab4363896525cf62 P b2339d533f554ac98c7cb0df03363ae6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 20:27:33.507712 4505 log.cc:1196] T e09a5f59ed39486fab4363896525cf62 P b2339d533f554ac98c7cb0df03363ae6: Deleting WAL directory at /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790713339161492-17621-0/minicluster-data/ts-2/wal/wals/e09a5f59ed39486fab4363896525cf62
I20260929 20:27:33.508220 4505 ts_tablet_manager.cc:1939] T e09a5f59ed39486fab4363896525cf62 P b2339d533f554ac98c7cb0df03363ae6: Deleting consensus metadata
I20260929 20:27:33.509812 4002 catalog_manager.cc:4841] TS b2339d533f554ac98c7cb0df03363ae6 (127.17.53.67:43643): tablet e09a5f59ed39486fab4363896525cf62 (table test-table [id=5f96ad4d38304764897a4830b09b4949]) successfully deleted
I20260929 20:27:33.612072 17621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu with pid 4075
W20260929 20:27:33.630323 4003 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.17.53.65:45935 (error 108)
I20260929 20:27:33.630669 17621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu pid 4207
I20260929 20:27:33.632575 4480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:27:33.631997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.65:45935 (local address 127.0.0.1:60856)
0929 20:27:33.632222 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0929 20:27:33.632392 (+ 170us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.53.65:45935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260929 20:27:33.632838 4003 catalog_manager.cc:4584] TS 2eb6300c78c4445e9c3d409a041c1a56 (127.17.53.65:45935): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e09a5f59ed39486fab4363896525cf62: Network error: Client connection negotiation failed: client connection to 127.17.53.65:45935: connect: Connection refused (error 111)
I20260929 20:27:33.638427 4481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:27:33.631547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34280 (local address 127.17.53.66:37193)
0929 20:27:33.631738 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0929 20:27:33.631745 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:27:33.631840 (+ 95us) server_negotiation.cc:407] Connection header received
0929 20:27:33.632081 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:27:33.632087 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:27:33.632209 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:27:33.632319 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:27:33.632822 (+ 503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:27:33.633637 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:27:33.635530 (+ 1893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:27:33.635913 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:27:33.636011 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:27:33.637109 (+ 1098us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:27:33.637118 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:27:33.637133 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:27:33.637717 (+ 584us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:27:33.637765 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:27:33.637784 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:27:33.637868 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0929 20:27:33.638022 (+ 154us) server_negotiation.cc:299] Negotiation successful
0929 20:27:33.638110 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 20:27:33.639122 4329 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:27:33.686682 4480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:27:33.686142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.65:45935 (local address 127.0.0.1:60872)
0929 20:27:33.686382 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0929 20:27:33.686489 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.53.65:45935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260929 20:27:33.687963 4480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:27:33.687546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.65:45935 (local address 127.0.0.1:60888)
0929 20:27:33.687745 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0929 20:27:33.687823 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.53.65:45935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106}
I20260929 20:27:33.728828 17621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu with pid 4207
I20260929 20:27:33.748600 4480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:27:33.747983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.65:45935 (local address 127.0.0.1:60900)
0929 20:27:33.748243 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0929 20:27:33.748358 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.53.65:45935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260929 20:27:33.748648 17621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu pid 4337
I20260929 20:27:33.749908 4480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:27:33.749321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.65:45935 (local address 127.0.0.1:60916)
0929 20:27:33.749680 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0929 20:27:33.749769 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.53.65:45935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":243}
I20260929 20:27:33.756112 4479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:27:33.749495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46412 (local address 127.17.53.67:43643)
0929 20:27:33.749682 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0929 20:27:33.749690 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 20:27:33.749741 (+ 51us) server_negotiation.cc:407] Connection header received
0929 20:27:33.749979 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:27:33.749986 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:27:33.750135 (+ 149us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:27:33.750251 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:27:33.750647 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:27:33.751506 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:27:33.752954 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:27:33.753395 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:27:33.753558 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:27:33.754601 (+ 1043us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:27:33.754610 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:27:33.754624 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:27:33.755333 (+ 709us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:27:33.755379 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:27:33.755402 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:27:33.755547 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0929 20:27:33.755696 (+ 149us) server_negotiation.cc:299] Negotiation successful
0929 20:27:33.755790 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260929 20:27:33.756827 4459 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:27:33.830461 4480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:27:33.830031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.65:45935 (local address 127.0.0.1:60918)
0929 20:27:33.830226 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0929 20:27:33.830313 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.53.65:45935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260929 20:27:33.831753 4480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:27:33.831374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.65:45935 (local address 127.0.0.1:60928)
0929 20:27:33.831573 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0929 20:27:33.831653 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.53.65:45935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260929 20:27:33.876809 17621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu with pid 4337
I20260929 20:27:33.896627 17621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu pid 3985
I20260929 20:27:33.897701 4046 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:27:34.006683 4480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:27:34.006032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.65:45935 (local address 127.0.0.1:60944)
0929 20:27:34.006274 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0929 20:27:34.006403 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.53.65:45935: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260929 20:27:34.006980 17621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu with pid 3985
2026-09-29T20:27:34Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (2821 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790713339161492-17621-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 20:27:34 dist-test-slave-zbnc krb5kdc[4516](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 20:27:34 dist-test-slave-zbnc krb5kdc[4516](info): set up 2 sockets
Sep 29 20:27:34 dist-test-slave-zbnc krb5kdc[4516](info): commencing operation
krb5kdc: starting...
W20260929 20:27:35.120772 17621 mini_kdc.cc:121] Time spent starting KDC: real 1.072s user 0.000s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790713339161492-17621-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790713339161492-17621-0/krb5kdc/test-user.keytab.
Sep 29 20:27:35 dist-test-slave-zbnc krb5kdc[4516](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713655, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T20:27:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T20:27:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.17.53.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.53.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.53.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.126.keytab.
Entry for principal kudu/127.17.53.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.126.keytab.
I20260929 20:27:35.370847 17621 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
/tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790713339161492-17621-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790713339161492-17621-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790713339161492-17621-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790713339161492-17621-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.53.126:44651
--webserver_interface=127.17.53.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.126.keytab
--principal=kudu/127.17.53.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.53.84:36603
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.53.126:44651
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790713339161492-17621-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790713339161492-17621-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790713339161492-17621-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:27:35.707878 4529 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:27:35.708213 4529 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:27:35.708376 4529 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:27:35.715669 4529 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:27:35.715811 4529 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:27:35.715900 4529 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:27:35.715999 4529 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:27:35.716080 4529 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:27:35.725847 4529 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.53.84:36603
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790713339161492-17621-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790713339161492-17621-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.53.126:44651
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.126.keytab
--principal=kudu/127.17.53.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.53.126:44651
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790713339161492-17621-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.53.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790713339161492-17621-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:27:35.727221 4529 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:27:35.728624 4529 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:27:35.740904 4534 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:27:35.740974 4537 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:27:35.740937 4535 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:27:35.741814 4529 server_base.cc:1034] running on GCE node
I20260929 20:27:35.742602 4529 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:27:35.744269 4529 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:27:35.745501 4529 hybrid_clock.cc:648] HybridClock initialized: now 1790713655745436 us; error 37 us; skew 500 ppm
I20260929 20:27:35.745883 4529 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:27:35 dist-test-slave-zbnc krb5kdc[4516](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713655, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.53.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:27:35.753512 4529 init.cc:377] Logged in from keytab as kudu/127.17.53.126@KRBTEST.COM (short username kudu)
I20260929 20:27:35.755270 4529 webserver.cc:460] Webserver started at http://127.17.53.126:37549/ using document root <none> and password file <none>
I20260929 20:27:35.756016 4529 fs_manager.cc:359] Metadata directory not provided
I20260929 20:27:35.756163 4529 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:27:35.756475 4529 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:27:35.759644 4529 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790713339161492-17621-0/minicluster-data/master-0/data/instance:
uuid: "79e73df628da45d2bea460ff89bcef8b"
format_stamp: "Formatted at 2026-09-29 20:27:35 on dist-test-slave-zbnc"
server_key: "586aaa7849e4332dc2a2d616107d435d"
server_key_iv: "9f2b1a43ee0d780b39a0d9a7abb5c095"
server_key_version: "encryptionkey@0"
I20260929 20:27:35.760425 4529 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790713339161492-17621-0/minicluster-data/master-0/wal/instance:
uuid: "79e73df628da45d2bea460ff89bcef8b"
format_stamp: "Formatted at 2026-09-29 20:27:35 on dist-test-slave-zbnc"
server_key: "586aaa7849e4332dc2a2d616107d435d"
server_key_iv: "9f2b1a43ee0d780b39a0d9a7abb5c095"
server_key_version: "encryptionkey@0"
I20260929 20:27:35.765849 4529 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 20:27:35.769459 4544 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:27:35.770833 4529 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 20:27:35.771113 4529 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790713339161492-17621-0/minicluster-data/master-0/data,/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790713339161492-17621-0/minicluster-data/master-0/wal
uuid: "79e73df628da45d2bea460ff89bcef8b"
format_stamp: "Formatted at 2026-09-29 20:27:35 on dist-test-slave-zbnc"
server_key: "586aaa7849e4332dc2a2d616107d435d"
server_key_iv: "9f2b1a43ee0d780b39a0d9a7abb5c095"
server_key_version: "encryptionkey@0"
I20260929 20:27:35.771319 4529 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790713339161492-17621-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790713339161492-17621-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790713339161492-17621-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:27:35.787541 4529 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:27:35.791725 4529 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:27:35.792025 4529 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:27:35.810779 4529 rpc_server.cc:307] RPC server started. Bound to: 127.17.53.126:44651
I20260929 20:27:35.810806 4595 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.53.126:44651 every 8 connection(s)
I20260929 20:27:35.812325 4529 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790713339161492-17621-0/minicluster-data/master-0/data/info.pb
I20260929 20:27:35.816504 4596 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:27:35.816470 17621 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu as pid 4529
I20260929 20:27:35.816668 17621 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790713339161492-17621-0/minicluster-data/master-0/wal/instance
I20260929 20:27:35.817260 17621 external_mini_cluster.cc:1428] Setting key 7240805263ce1907e888fc3c3a576977
I20260929 20:27:35.831722 4596 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 79e73df628da45d2bea460ff89bcef8b: Bootstrap starting.
Sep 29 20:27:35 dist-test-slave-zbnc krb5kdc[4516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713655, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.53.126@KRBTEST.COM
I20260929 20:27:35.834998 4596 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 79e73df628da45d2bea460ff89bcef8b: Neither blocks nor log segments found. Creating new log.
I20260929 20:27:35.836021 4596 log.cc:824] T 00000000000000000000000000000000 P 79e73df628da45d2bea460ff89bcef8b: Log is configured to *not* fsync() on all Append() calls
I20260929 20:27:35.838932 4596 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 79e73df628da45d2bea460ff89bcef8b: No bootstrap required, opened a new log
I20260929 20:27:35.841336 4599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:27:35.819202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59894 (local address 127.17.53.126:44651)
0929 20:27:35.820055 (+ 853us) server_negotiation.cc:206] Beginning negotiation
0929 20:27:35.820070 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 20:27:35.820115 (+ 45us) server_negotiation.cc:407] Connection header received
0929 20:27:35.821758 (+ 1643us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:27:35.821781 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:27:35.822195 (+ 414us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:27:35.822559 (+ 364us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:27:35.824441 (+ 1882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:27:35.825736 (+ 1295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:27:35.827579 (+ 1843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:27:35.828292 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:27:35.828426 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:27:35.833411 (+ 4985us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:27:35.833419 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:27:35.833434 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:27:35.833480 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:27:35.836635 (+ 3155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:27:35.837445 (+ 810us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:27:35.837452 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:27:35.837459 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:27:35.837631 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:27:35.838049 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:27:35.838061 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:27:35.838063 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:27:35.838591 (+ 528us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:27:35.838922 (+ 331us) server_negotiation.cc:1035] Waiting for connection context
0929 20:27:35.839441 (+ 519us) server_negotiation.cc:299] Negotiation successful
0929 20:27:35.839942 (+ 501us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":645,"thread_start_us":420,"threads_started":1}
I20260929 20:27:35.847470 4596 raft_consensus.cc:348] T 00000000000000000000000000000000 P 79e73df628da45d2bea460ff89bcef8b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79e73df628da45d2bea460ff89bcef8b" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 44651 } }
I20260929 20:27:35.847832 4596 raft_consensus.cc:374] T 00000000000000000000000000000000 P 79e73df628da45d2bea460ff89bcef8b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:27:35.847903 4596 raft_consensus.cc:729] T 00000000000000000000000000000000 P 79e73df628da45d2bea460ff89bcef8b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79e73df628da45d2bea460ff89bcef8b, State: Initialized, Role: FOLLOWER
I20260929 20:27:35.848408 4596 consensus_queue.cc:260] T 00000000000000000000000000000000 P 79e73df628da45d2bea460ff89bcef8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79e73df628da45d2bea460ff89bcef8b" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 44651 } }
I20260929 20:27:35.848608 4596 raft_consensus.cc:388] T 00000000000000000000000000000000 P 79e73df628da45d2bea460ff89bcef8b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 20:27:35.848716 4596 raft_consensus.cc:482] T 00000000000000000000000000000000 P 79e73df628da45d2bea460ff89bcef8b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 20:27:35.848841 4596 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 79e73df628da45d2bea460ff89bcef8b [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:27:35.851086 4596 raft_consensus.cc:504] T 00000000000000000000000000000000 P 79e73df628da45d2bea460ff89bcef8b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79e73df628da45d2bea460ff89bcef8b" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 44651 } }
I20260929 20:27:35.851647 4596 leader_election.cc:302] T 00000000000000000000000000000000 P 79e73df628da45d2bea460ff89bcef8b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 79e73df628da45d2bea460ff89bcef8b; no voters:
I20260929 20:27:35.852212 4596 leader_election.cc:288] T 00000000000000000000000000000000 P 79e73df628da45d2bea460ff89bcef8b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 20:27:35.852401 4601 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 79e73df628da45d2bea460ff89bcef8b [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:27:35.853181 4601 raft_consensus.cc:686] T 00000000000000000000000000000000 P 79e73df628da45d2bea460ff89bcef8b [term 1 LEADER]: Becoming Leader. State: Replica: 79e73df628da45d2bea460ff89bcef8b, State: Running, Role: LEADER
I20260929 20:27:35.853745 4601 consensus_queue.cc:237] T 00000000000000000000000000000000 P 79e73df628da45d2bea460ff89bcef8b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79e73df628da45d2bea460ff89bcef8b" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 44651 } }
I20260929 20:27:35.854454 4596 sys_catalog.cc:564] T 00000000000000000000000000000000 P 79e73df628da45d2bea460ff89bcef8b [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:27:35.856916 4602 sys_catalog.cc:455] T 00000000000000000000000000000000 P 79e73df628da45d2bea460ff89bcef8b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "79e73df628da45d2bea460ff89bcef8b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79e73df628da45d2bea460ff89bcef8b" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 44651 } } }
I20260929 20:27:35.856902 4603 sys_catalog.cc:455] T 00000000000000000000000000000000 P 79e73df628da45d2bea460ff89bcef8b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 79e73df628da45d2bea460ff89bcef8b. Latest consensus state: current_term: 1 leader_uuid: "79e73df628da45d2bea460ff89bcef8b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79e73df628da45d2bea460ff89bcef8b" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 44651 } } }
I20260929 20:27:35.857468 4602 sys_catalog.cc:458] T 00000000000000000000000000000000 P 79e73df628da45d2bea460ff89bcef8b [sys.catalog]: This master's current role is: LEADER
I20260929 20:27:35.857465 4603 sys_catalog.cc:458] T 00000000000000000000000000000000 P 79e73df628da45d2bea460ff89bcef8b [sys.catalog]: This master's current role is: LEADER
I20260929 20:27:35.858462 4609 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:27:35.868155 4609 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:27:35.877611 4609 catalog_manager.cc:1347] Generated new cluster ID: 2f3bc88f2ff84bd39f266f44a19e2073
I20260929 20:27:35.877745 4609 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:27:35.887946 4609 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 20:27:35.889451 4609 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:27:35.897522 4609 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 79e73df628da45d2bea460ff89bcef8b: Generated new TSK 0
I20260929 20:27:35.898381 4609 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.17.53.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.53.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.53.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.65.keytab.
Entry for principal kudu/127.17.53.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.65.keytab.
I20260929 20:27:35.955282 17621 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
/tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790713339161492-17621-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790713339161492-17621-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790713339161492-17621-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790713339161492-17621-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.53.65:0
--local_ip_for_outbound_sockets=127.17.53.65
--webserver_interface=127.17.53.65
--webserver_port=0
--tserver_master_addrs=127.17.53.126:44651
--keytab_file=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.65.keytab
--principal=kudu/127.17.53.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.53.84:36603
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790713339161492-17621-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790713339161492-17621-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790713339161492-17621-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:27:36.296103 4622 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:27:36.296394 4622 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:27:36.296566 4622 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:27:36.304133 4622 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:27:36.304258 4622 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:27:36.304430 4622 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.53.65
I20260929 20:27:36.314254 4622 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.53.84:36603
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790713339161492-17621-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790713339161492-17621-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.65.keytab
--principal=kudu/127.17.53.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.53.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790713339161492-17621-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.17.53.65
--webserver_port=0
--tserver_master_addrs=127.17.53.126:44651
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.53.65
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790713339161492-17621-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:27:36.315730 4622 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:27:36.317129 4622 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:27:36.330271 4627 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:27:36.330289 4628 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:27:36.330313 4630 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:27:36.331364 4622 server_base.cc:1034] running on GCE node
I20260929 20:27:36.331962 4622 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:27:36.332930 4622 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:27:36.334144 4622 hybrid_clock.cc:648] HybridClock initialized: now 1790713656334081 us; error 51 us; skew 500 ppm
I20260929 20:27:36.334430 4622 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:27:36 dist-test-slave-zbnc krb5kdc[4516](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713656, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.53.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:27:36.337821 4622 init.cc:377] Logged in from keytab as kudu/127.17.53.65@KRBTEST.COM (short username kudu)
I20260929 20:27:36.339393 4622 webserver.cc:460] Webserver started at http://127.17.53.65:42897/ using document root <none> and password file <none>
I20260929 20:27:36.340297 4622 fs_manager.cc:359] Metadata directory not provided
I20260929 20:27:36.340440 4622 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:27:36.340803 4622 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:27:36.344307 4622 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790713339161492-17621-0/minicluster-data/ts-0/data/instance:
uuid: "0f75989f4187448e90cfb9179cd75ae9"
format_stamp: "Formatted at 2026-09-29 20:27:36 on dist-test-slave-zbnc"
server_key: "1e47948ec2dce1dfe4f295e86ecec39d"
server_key_iv: "ed7048e8a4a87352e9200c363324c5fe"
server_key_version: "encryptionkey@0"
I20260929 20:27:36.345041 4622 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790713339161492-17621-0/minicluster-data/ts-0/wal/instance:
uuid: "0f75989f4187448e90cfb9179cd75ae9"
format_stamp: "Formatted at 2026-09-29 20:27:36 on dist-test-slave-zbnc"
server_key: "1e47948ec2dce1dfe4f295e86ecec39d"
server_key_iv: "ed7048e8a4a87352e9200c363324c5fe"
server_key_version: "encryptionkey@0"
I20260929 20:27:36.350239 4622 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 20:27:36.353665 4637 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:27:36.355195 4622 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 20:27:36.355356 4622 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790713339161492-17621-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790713339161492-17621-0/minicluster-data/ts-0/wal
uuid: "0f75989f4187448e90cfb9179cd75ae9"
format_stamp: "Formatted at 2026-09-29 20:27:36 on dist-test-slave-zbnc"
server_key: "1e47948ec2dce1dfe4f295e86ecec39d"
server_key_iv: "ed7048e8a4a87352e9200c363324c5fe"
server_key_version: "encryptionkey@0"
I20260929 20:27:36.355569 4622 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790713339161492-17621-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790713339161492-17621-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790713339161492-17621-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:27:36.370699 4622 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:27:36.374603 4622 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:27:36.374945 4622 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:27:36.375924 4622 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:27:36.377301 4622 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:27:36.377377 4622 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:27:36.377537 4622 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:27:36.377586 4622 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:27:36.408943 4622 rpc_server.cc:307] RPC server started. Bound to: 127.17.53.65:46007
I20260929 20:27:36.409030 4749 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.53.65:46007 every 8 connection(s)
I20260929 20:27:36.410317 4622 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790713339161492-17621-0/minicluster-data/ts-0/data/info.pb
I20260929 20:27:36.411168 17621 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu as pid 4622
I20260929 20:27:36.411379 17621 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790713339161492-17621-0/minicluster-data/ts-0/wal/instance
I20260929 20:27:36.411857 17621 external_mini_cluster.cc:1428] Setting key 346dbea4e8f6cbf5ced8bfc244e4e9b7
Sep 29 20:27:36 dist-test-slave-zbnc krb5kdc[4516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713656, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.53.65@KRBTEST.COM for kudu/127.17.53.126@KRBTEST.COM
I20260929 20:27:36.433540 4752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:27:36.413034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.53.65:55279 (local address 127.17.53.126:44651)
0929 20:27:36.413508 (+ 474us) server_negotiation.cc:206] Beginning negotiation
0929 20:27:36.413514 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:27:36.415014 (+ 1500us) server_negotiation.cc:407] Connection header received
0929 20:27:36.417146 (+ 2132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:27:36.417154 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:27:36.417252 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:27:36.417375 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:27:36.419078 (+ 1703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:27:36.420287 (+ 1209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:27:36.422082 (+ 1795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:27:36.422493 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:27:36.422596 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:27:36.427484 (+ 4888us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:27:36.427490 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:27:36.427493 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:27:36.427529 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:27:36.430167 (+ 2638us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:27:36.431135 (+ 968us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:27:36.431139 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:27:36.431141 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:27:36.431262 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:27:36.431775 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:27:36.431778 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:27:36.431780 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:27:36.432041 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:27:36.432146 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0929 20:27:36.433102 (+ 956us) server_negotiation.cc:299] Negotiation successful
0929 20:27:36.433270 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":339,"thread_start_us":250,"threads_started":1}
I20260929 20:27:36.435233 4753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:27:36.413837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:44651 (local address 127.17.53.65:55279)
0929 20:27:36.414810 (+ 973us) negotiation.cc:107] Waiting for socket to connect
0929 20:27:36.414897 (+ 87us) client_negotiation.cc:174] Beginning negotiation
0929 20:27:36.416857 (+ 1960us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:27:36.417607 (+ 750us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:27:36.417616 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:27:36.418209 (+ 593us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:27:36.418836 (+ 627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:27:36.418909 (+ 73us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:27:36.420471 (+ 1562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:27:36.420476 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:27:36.421919 (+ 1443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:27:36.421940 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:27:36.422753 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:27:36.422764 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:27:36.423173 (+ 409us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:27:36.424022 (+ 849us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:27:36.424058 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:27:36.427224 (+ 3166us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:27:36.430324 (+ 3100us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:27:36.430336 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:27:36.430358 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:27:36.431017 (+ 659us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:27:36.431368 (+ 351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:27:36.431375 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:27:36.431382 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:27:36.431668 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:27:36.432169 (+ 501us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:27:36.432182 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:27:36.432696 (+ 514us) client_negotiation.cc:769] Sending connection context
0929 20:27:36.433011 (+ 315us) client_negotiation.cc:240] Negotiation successful
0929 20:27:36.433626 (+ 615us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":626,"thread_start_us":406,"threads_started":1}
I20260929 20:27:36.436985 4750 heartbeater.cc:347] Connected to a master server at 127.17.53.126:44651
I20260929 20:27:36.437345 4750 heartbeater.cc:464] Registering TS with master...
I20260929 20:27:36.438082 4750 heartbeater.cc:511] Master 127.17.53.126:44651 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.17.53.66@KRBTEST.COM; defaulting to no policy
I20260929 20:27:36.440318 4561 ts_manager.cc:194] Registered new tserver with Master: 0f75989f4187448e90cfb9179cd75ae9 (127.17.53.65:46007)
I20260929 20:27:36.441861 4561 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.53.65@KRBTEST.COM'} at 127.17.53.65:55279
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.53.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.53.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.66.keytab.
Entry for principal kudu/127.17.53.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.66.keytab.
I20260929 20:27:36.471938 17621 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
/tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790713339161492-17621-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790713339161492-17621-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790713339161492-17621-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790713339161492-17621-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.53.66:0
--local_ip_for_outbound_sockets=127.17.53.66
--webserver_interface=127.17.53.66
--webserver_port=0
--tserver_master_addrs=127.17.53.126:44651
--keytab_file=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.66.keytab
--principal=kudu/127.17.53.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.53.84:36603
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790713339161492-17621-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790713339161492-17621-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790713339161492-17621-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:27:36.802703 4756 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:27:36.803114 4756 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:27:36.803226 4756 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:27:36.810901 4756 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:27:36.811034 4756 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:27:36.811218 4756 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.53.66
I20260929 20:27:36.821025 4756 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.53.84:36603
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790713339161492-17621-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790713339161492-17621-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.66.keytab
--principal=kudu/127.17.53.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.53.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790713339161492-17621-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.17.53.66
--webserver_port=0
--tserver_master_addrs=127.17.53.126:44651
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.53.66
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790713339161492-17621-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:27:36.822229 4756 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:27:36.823587 4756 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:27:36.836241 4761 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:27:36.836287 4762 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:27:36.836397 4764 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:27:36.837333 4756 server_base.cc:1034] running on GCE node
I20260929 20:27:36.837918 4756 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:27:36.838850 4756 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:27:36.840118 4756 hybrid_clock.cc:648] HybridClock initialized: now 1790713656840076 us; error 34 us; skew 500 ppm
I20260929 20:27:36.840433 4756 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:27:36 dist-test-slave-zbnc krb5kdc[4516](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713656, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.53.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:27:36.843446 4756 init.cc:377] Logged in from keytab as kudu/127.17.53.66@KRBTEST.COM (short username kudu)
I20260929 20:27:36.844822 4756 webserver.cc:460] Webserver started at http://127.17.53.66:34621/ using document root <none> and password file <none>
I20260929 20:27:36.845454 4756 fs_manager.cc:359] Metadata directory not provided
I20260929 20:27:36.845551 4756 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:27:36.845800 4756 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:27:36.848757 4756 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790713339161492-17621-0/minicluster-data/ts-1/data/instance:
uuid: "9e75f9ac04d6463b8fc109218fdd82f3"
format_stamp: "Formatted at 2026-09-29 20:27:36 on dist-test-slave-zbnc"
server_key: "1cb35b633b602664a5b903157ea0cd5a"
server_key_iv: "9c780057078a847e84c9838ca610ab44"
server_key_version: "encryptionkey@0"
I20260929 20:27:36.849457 4756 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790713339161492-17621-0/minicluster-data/ts-1/wal/instance:
uuid: "9e75f9ac04d6463b8fc109218fdd82f3"
format_stamp: "Formatted at 2026-09-29 20:27:36 on dist-test-slave-zbnc"
server_key: "1cb35b633b602664a5b903157ea0cd5a"
server_key_iv: "9c780057078a847e84c9838ca610ab44"
server_key_version: "encryptionkey@0"
I20260929 20:27:36.854434 4756 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 20:27:36.857844 4771 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:27:36.859443 4756 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260929 20:27:36.859591 4756 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790713339161492-17621-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790713339161492-17621-0/minicluster-data/ts-1/wal
uuid: "9e75f9ac04d6463b8fc109218fdd82f3"
format_stamp: "Formatted at 2026-09-29 20:27:36 on dist-test-slave-zbnc"
server_key: "1cb35b633b602664a5b903157ea0cd5a"
server_key_iv: "9c780057078a847e84c9838ca610ab44"
server_key_version: "encryptionkey@0"
I20260929 20:27:36.859776 4756 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790713339161492-17621-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790713339161492-17621-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790713339161492-17621-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:27:36.881161 4756 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:27:36.885228 4756 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:27:36.885514 4756 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:27:36.886479 4756 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:27:36.887847 4756 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:27:36.887919 4756 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:27:36.888068 4756 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:27:36.888131 4756 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:27:36.921883 4756 rpc_server.cc:307] RPC server started. Bound to: 127.17.53.66:43455
I20260929 20:27:36.921911 4883 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.53.66:43455 every 8 connection(s)
I20260929 20:27:36.923305 4756 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790713339161492-17621-0/minicluster-data/ts-1/data/info.pb
I20260929 20:27:36.926220 17621 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu as pid 4756
I20260929 20:27:36.926404 17621 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790713339161492-17621-0/minicluster-data/ts-1/wal/instance
I20260929 20:27:36.927040 17621 external_mini_cluster.cc:1428] Setting key 36997149114a0c4e8f93293f548ae770
Sep 29 20:27:36 dist-test-slave-zbnc krb5kdc[4516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713656, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.53.66@KRBTEST.COM for kudu/127.17.53.126@KRBTEST.COM
I20260929 20:27:36.947831 4752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:27:36.925954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.53.66:39717 (local address 127.17.53.126:44651)
0929 20:27:36.926131 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0929 20:27:36.926135 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:27:36.927695 (+ 1560us) server_negotiation.cc:407] Connection header received
0929 20:27:36.929978 (+ 2283us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:27:36.929990 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:27:36.930058 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:27:36.930134 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:27:36.932294 (+ 2160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:27:36.933032 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:27:36.935585 (+ 2553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:27:36.936255 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:27:36.936375 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:27:36.940714 (+ 4339us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:27:36.940723 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:27:36.940731 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:27:36.940782 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:27:36.944411 (+ 3629us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:27:36.945189 (+ 778us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:27:36.945197 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:27:36.945203 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:27:36.945369 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:27:36.945773 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:27:36.945780 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:27:36.945786 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:27:36.946198 (+ 412us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:27:36.946360 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0929 20:27:36.947099 (+ 739us) server_negotiation.cc:299] Negotiation successful
0929 20:27:36.947373 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260929 20:27:36.948652 4886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:27:36.926693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:44651 (local address 127.17.53.66:39717)
0929 20:27:36.927527 (+ 834us) negotiation.cc:107] Waiting for socket to connect
0929 20:27:36.927572 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0929 20:27:36.929648 (+ 2076us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:27:36.930405 (+ 757us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:27:36.930419 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:27:36.931287 (+ 868us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:27:36.932123 (+ 836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:27:36.932152 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:27:36.933192 (+ 1040us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:27:36.933200 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:27:36.935440 (+ 2240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:27:36.935459 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:27:36.936650 (+ 1191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:27:36.936658 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:27:36.936974 (+ 316us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:27:36.937754 (+ 780us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:27:36.937787 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:27:36.940440 (+ 2653us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:27:36.944571 (+ 4131us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:27:36.944579 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:27:36.944594 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:27:36.945050 (+ 456us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:27:36.945474 (+ 424us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:27:36.945477 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:27:36.945480 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:27:36.945661 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:27:36.946315 (+ 654us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:27:36.946322 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:27:36.946664 (+ 342us) client_negotiation.cc:769] Sending connection context
0929 20:27:36.946968 (+ 304us) client_negotiation.cc:240] Negotiation successful
0929 20:27:36.947449 (+ 481us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":579,"thread_start_us":364,"threads_started":1}
I20260929 20:27:36.950228 4884 heartbeater.cc:347] Connected to a master server at 127.17.53.126:44651
I20260929 20:27:36.950557 4884 heartbeater.cc:464] Registering TS with master...
I20260929 20:27:36.951349 4884 heartbeater.cc:511] Master 127.17.53.126:44651 requested a full tablet report, sending...
I20260929 20:27:36.952912 4561 ts_manager.cc:194] Registered new tserver with Master: 9e75f9ac04d6463b8fc109218fdd82f3 (127.17.53.66:43455)
I20260929 20:27:36.953719 4561 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.53.66@KRBTEST.COM'} at 127.17.53.66:39717
WARNING: no policy specified for kudu/127.17.53.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.53.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.53.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.67.keytab.
Entry for principal kudu/127.17.53.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.67.keytab.
I20260929 20:27:36.989821 17621 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
/tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790713339161492-17621-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790713339161492-17621-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790713339161492-17621-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790713339161492-17621-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.53.67:0
--local_ip_for_outbound_sockets=127.17.53.67
--webserver_interface=127.17.53.67
--webserver_port=0
--tserver_master_addrs=127.17.53.126:44651
--keytab_file=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.67.keytab
--principal=kudu/127.17.53.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.53.84:36603
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790713339161492-17621-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790713339161492-17621-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790713339161492-17621-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:27:37.324438 4889 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:27:37.324800 4889 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:27:37.325019 4889 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:27:37.332885 4889 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:27:37.333052 4889 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:27:37.333233 4889 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.53.67
I20260929 20:27:37.343416 4889 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.53.84:36603
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790713339161492-17621-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790713339161492-17621-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.67.keytab
--principal=kudu/127.17.53.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.53.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790713339161492-17621-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.17.53.67
--webserver_port=0
--tserver_master_addrs=127.17.53.126:44651
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.53.67
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790713339161492-17621-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:27:37.344803 4889 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:27:37.346691 4889 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:27:37.360776 4895 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:27:37.360817 4897 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:27:37.360734 4894 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:27:37.361977 4889 server_base.cc:1034] running on GCE node
I20260929 20:27:37.362645 4889 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:27:37.363857 4889 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:27:37.365090 4889 hybrid_clock.cc:648] HybridClock initialized: now 1790713657364985 us; error 83 us; skew 500 ppm
I20260929 20:27:37.365417 4889 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:27:37 dist-test-slave-zbnc krb5kdc[4516](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713657, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.53.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:27:37.368666 4889 init.cc:377] Logged in from keytab as kudu/127.17.53.67@KRBTEST.COM (short username kudu)
I20260929 20:27:37.370190 4889 webserver.cc:460] Webserver started at http://127.17.53.67:41681/ using document root <none> and password file <none>
I20260929 20:27:37.370960 4889 fs_manager.cc:359] Metadata directory not provided
I20260929 20:27:37.371104 4889 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:27:37.371441 4889 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:27:37.374426 4889 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790713339161492-17621-0/minicluster-data/ts-2/data/instance:
uuid: "d3aeaaf23e884273a2d2fb55a07593c6"
format_stamp: "Formatted at 2026-09-29 20:27:37 on dist-test-slave-zbnc"
server_key: "2cc5239b8fd7dccb73b640ee643adb30"
server_key_iv: "eab89f6c4e1e9753bfb8d73980ef3717"
server_key_version: "encryptionkey@0"
I20260929 20:27:37.375268 4889 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790713339161492-17621-0/minicluster-data/ts-2/wal/instance:
uuid: "d3aeaaf23e884273a2d2fb55a07593c6"
format_stamp: "Formatted at 2026-09-29 20:27:37 on dist-test-slave-zbnc"
server_key: "2cc5239b8fd7dccb73b640ee643adb30"
server_key_iv: "eab89f6c4e1e9753bfb8d73980ef3717"
server_key_version: "encryptionkey@0"
I20260929 20:27:37.380419 4889 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260929 20:27:37.383759 4904 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:27:37.385085 4889 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 20:27:37.385254 4889 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790713339161492-17621-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790713339161492-17621-0/minicluster-data/ts-2/wal
uuid: "d3aeaaf23e884273a2d2fb55a07593c6"
format_stamp: "Formatted at 2026-09-29 20:27:37 on dist-test-slave-zbnc"
server_key: "2cc5239b8fd7dccb73b640ee643adb30"
server_key_iv: "eab89f6c4e1e9753bfb8d73980ef3717"
server_key_version: "encryptionkey@0"
I20260929 20:27:37.385459 4889 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790713339161492-17621-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790713339161492-17621-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790713339161492-17621-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:27:37.409881 4889 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:27:37.414304 4889 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:27:37.414602 4889 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:27:37.415593 4889 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:27:37.416958 4889 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:27:37.417060 4889 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:27:37.417210 4889 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:27:37.417289 4889 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:27:37.444941 4750 heartbeater.cc:503] Master 127.17.53.126:44651 was elected leader, sending a full tablet report...
I20260929 20:27:37.451050 4889 rpc_server.cc:307] RPC server started. Bound to: 127.17.53.67:44117
I20260929 20:27:37.451133 5016 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.53.67:44117 every 8 connection(s)
I20260929 20:27:37.452346 4889 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790713339161492-17621-0/minicluster-data/ts-2/data/info.pb
I20260929 20:27:37.454988 17621 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu as pid 4889
I20260929 20:27:37.455209 17621 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790713339161492-17621-0/minicluster-data/ts-2/wal/instance
I20260929 20:27:37.455971 17621 external_mini_cluster.cc:1428] Setting key 06ef09b1a5fdf6e1599c6ac44e10f11a
Sep 29 20:27:37 dist-test-slave-zbnc krb5kdc[4516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713657, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.53.67@KRBTEST.COM for kudu/127.17.53.126@KRBTEST.COM
I20260929 20:27:37.473754 5019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:27:37.454960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.53.67:43485 (local address 127.17.53.126:44651)
0929 20:27:37.459331 (+ 4371us) server_negotiation.cc:206] Beginning negotiation
0929 20:27:37.459338 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:27:37.459360 (+ 22us) server_negotiation.cc:407] Connection header received
0929 20:27:37.459468 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:27:37.459474 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:27:37.459590 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:27:37.459719 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:27:37.461239 (+ 1520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:27:37.462039 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:27:37.463699 (+ 1660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:27:37.464171 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:27:37.464306 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:27:37.467674 (+ 3368us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:27:37.467679 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:27:37.467682 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:27:37.467717 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:27:37.470264 (+ 2547us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:27:37.471240 (+ 976us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:27:37.471246 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:27:37.471249 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:27:37.471399 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:27:37.471861 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:27:37.471866 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:27:37.471869 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:27:37.472232 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:27:37.472399 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0929 20:27:37.473140 (+ 741us) server_negotiation.cc:299] Negotiation successful
0929 20:27:37.473369 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4217,"thread_start_us":4099,"threads_started":1}
I20260929 20:27:37.474952 5020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:27:37.455632 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:44651 (local address 127.17.53.67:43485)
0929 20:27:37.456573 (+ 941us) negotiation.cc:107] Waiting for socket to connect
0929 20:27:37.456627 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0929 20:27:37.459004 (+ 2377us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:27:37.459942 (+ 938us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:27:37.459951 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:27:37.460513 (+ 562us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:27:37.461091 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:27:37.461107 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:27:37.462170 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:27:37.462174 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:27:37.463560 (+ 1386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:27:37.463573 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:27:37.464401 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:27:37.464406 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:27:37.464604 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:27:37.465206 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:27:37.465228 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:27:37.467460 (+ 2232us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:27:37.470437 (+ 2977us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:27:37.470447 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:27:37.470465 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:27:37.471112 (+ 647us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:27:37.471528 (+ 416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:27:37.471540 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:27:37.471543 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:27:37.471739 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:27:37.472376 (+ 637us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:27:37.472385 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:27:37.472757 (+ 372us) client_negotiation.cc:769] Sending connection context
0929 20:27:37.473024 (+ 267us) client_negotiation.cc:240] Negotiation successful
0929 20:27:37.473600 (+ 576us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":639,"thread_start_us":423,"threads_started":1}
I20260929 20:27:37.476310 5017 heartbeater.cc:347] Connected to a master server at 127.17.53.126:44651
I20260929 20:27:37.476635 5017 heartbeater.cc:464] Registering TS with master...
I20260929 20:27:37.477353 5017 heartbeater.cc:511] Master 127.17.53.126:44651 requested a full tablet report, sending...
I20260929 20:27:37.478817 4561 ts_manager.cc:194] Registered new tserver with Master: d3aeaaf23e884273a2d2fb55a07593c6 (127.17.53.67:44117)
I20260929 20:27:37.479715 4561 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.17.53.67@KRBTEST.COM'} at 127.17.53.67:43485
I20260929 20:27:37.484391 17621 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 20:27:37.500938 5019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:27:37.488639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59900 (local address 127.17.53.126:44651)
0929 20:27:37.488833 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0929 20:27:37.488840 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:27:37.489195 (+ 355us) server_negotiation.cc:407] Connection header received
0929 20:27:37.489323 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:27:37.489327 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:27:37.489412 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:27:37.489499 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:27:37.490967 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:27:37.491720 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:27:37.492967 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:27:37.493396 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:27:37.493496 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:27:37.495894 (+ 2398us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:27:37.495899 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:27:37.495902 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:27:37.495935 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:27:37.498554 (+ 2619us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:27:37.499209 (+ 655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:27:37.499213 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:27:37.499215 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:27:37.499359 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:27:37.499695 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:27:37.499698 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:27:37.499700 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:27:37.500003 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:27:37.500182 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0929 20:27:37.500469 (+ 287us) server_negotiation.cc:299] Negotiation successful
0929 20:27:37.500708 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260929 20:27:37.504911 4561 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59900:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 20:27:37.512025 4561 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 20:27:37.535408 5032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:27:37.524888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59170 (local address 127.17.53.67:44117)
0929 20:27:37.526898 (+ 2010us) server_negotiation.cc:206] Beginning negotiation
0929 20:27:37.526910 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 20:27:37.526954 (+ 44us) server_negotiation.cc:407] Connection header received
0929 20:27:37.527055 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:27:37.527063 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:27:37.527326 (+ 263us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:27:37.528253 (+ 927us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:27:37.528607 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:27:37.529987 (+ 1380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:27:37.532954 (+ 2967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:27:37.534011 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:27:37.534147 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:27:37.534400 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0929 20:27:37.534775 (+ 375us) server_negotiation.cc:299] Negotiation successful
0929 20:27:37.535010 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1703,"spinlock_wait_cycles":27392,"thread_start_us":1681,"threads_started":1}
I20260929 20:27:37.535430 5031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:27:37.524638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.67:44117 (local address 127.0.0.1:59170)
0929 20:27:37.525752 (+ 1114us) negotiation.cc:107] Waiting for socket to connect
0929 20:27:37.525803 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0929 20:27:37.526022 (+ 219us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:27:37.527598 (+ 1576us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:27:37.527607 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:27:37.527642 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:27:37.528047 (+ 405us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:27:37.528063 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:27:37.530125 (+ 2062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:27:37.530134 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:27:37.532786 (+ 2652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:27:37.532801 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:27:37.534199 (+ 1398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:27:37.534205 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:27:37.534492 (+ 287us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:27:37.534556 (+ 64us) client_negotiation.cc:769] Sending connection context
0929 20:27:37.534703 (+ 147us) client_negotiation.cc:240] Negotiation successful
0929 20:27:37.534921 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":788,"mutex_wait_us":142,"thread_start_us":2354,"threads_started":1}
I20260929 20:27:37.536295 5030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:27:37.524902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.66:43455 (local address 127.0.0.1:40598)
0929 20:27:37.527318 (+ 2416us) negotiation.cc:107] Waiting for socket to connect
0929 20:27:37.527340 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 20:27:37.527487 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:27:37.528624 (+ 1137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:27:37.528629 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:27:37.528661 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:27:37.529160 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:27:37.529171 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:27:37.531098 (+ 1927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:27:37.531105 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:27:37.533153 (+ 2048us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:27:37.533168 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:27:37.535371 (+ 2203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:27:37.535380 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:27:37.535618 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:27:37.535655 (+ 37us) client_negotiation.cc:769] Sending connection context
0929 20:27:37.535738 (+ 83us) client_negotiation.cc:240] Negotiation successful
0929 20:27:37.535866 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2265,"mutex_wait_us":2,"thread_start_us":859,"threads_started":1}
I20260929 20:27:37.536844 5033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:27:37.525510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40598 (local address 127.17.53.66:43455)
0929 20:27:37.528018 (+ 2508us) server_negotiation.cc:206] Beginning negotiation
0929 20:27:37.528028 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 20:27:37.528057 (+ 29us) server_negotiation.cc:407] Connection header received
0929 20:27:37.528153 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:27:37.528161 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:27:37.528387 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:27:37.529349 (+ 962us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:27:37.529600 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:27:37.530981 (+ 1381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:27:37.533319 (+ 2338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:27:37.534805 (+ 1486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:27:37.535012 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:27:37.535208 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0929 20:27:37.536124 (+ 916us) server_negotiation.cc:299] Negotiation successful
0929 20:27:37.536327 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2343,"thread_start_us":2225,"threads_started":1}
I20260929 20:27:37.538089 5035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:27:37.527996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.65:46007 (local address 127.0.0.1:49244)
0929 20:27:37.528816 (+ 820us) negotiation.cc:107] Waiting for socket to connect
0929 20:27:37.528832 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 20:27:37.529027 (+ 195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:27:37.529934 (+ 907us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:27:37.529940 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:27:37.529968 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:27:37.530392 (+ 424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:27:37.530403 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:27:37.532322 (+ 1919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:27:37.532327 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:27:37.534547 (+ 2220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:27:37.534562 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:27:37.537341 (+ 2779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:27:37.537346 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:27:37.537460 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:27:37.537481 (+ 21us) client_negotiation.cc:769] Sending connection context
0929 20:27:37.537584 (+ 103us) client_negotiation.cc:240] Negotiation successful
0929 20:27:37.537726 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":665,"mutex_wait_us":62,"thread_start_us":530,"threads_started":1}
I20260929 20:27:37.538133 5034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:27:37.527687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49244 (local address 127.17.53.65:46007)
0929 20:27:37.528368 (+ 681us) server_negotiation.cc:206] Beginning negotiation
0929 20:27:37.528376 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 20:27:37.528916 (+ 540us) server_negotiation.cc:407] Connection header received
0929 20:27:37.529224 (+ 308us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:27:37.529234 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:27:37.529465 (+ 231us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:27:37.529626 (+ 161us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:27:37.530582 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:27:37.532147 (+ 1565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:27:37.534716 (+ 2569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:27:37.536858 (+ 2142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:27:37.537003 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:27:37.537226 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0929 20:27:37.537631 (+ 405us) server_negotiation.cc:299] Negotiation successful
0929 20:27:37.537794 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":515,"thread_start_us":419,"threads_started":1}
I20260929 20:27:37.540432 4819 tablet_service.cc:1467] Processing CreateTablet for tablet 5a13f36c2b5d48599f2cf5b8a28fb4fe (DEFAULT_TABLE table=test-table [id=7d1a56cf18d74308a9117153bf5c4c49]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:27:37.540807 4952 tablet_service.cc:1467] Processing CreateTablet for tablet 5a13f36c2b5d48599f2cf5b8a28fb4fe (DEFAULT_TABLE table=test-table [id=7d1a56cf18d74308a9117153bf5c4c49]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:27:37.541990 4819 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a13f36c2b5d48599f2cf5b8a28fb4fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:27:37.542178 4952 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a13f36c2b5d48599f2cf5b8a28fb4fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:27:37.541853 4685 tablet_service.cc:1467] Processing CreateTablet for tablet 5a13f36c2b5d48599f2cf5b8a28fb4fe (DEFAULT_TABLE table=test-table [id=7d1a56cf18d74308a9117153bf5c4c49]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:27:37.543115 4685 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a13f36c2b5d48599f2cf5b8a28fb4fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:27:37.555301 5036 tablet_bootstrap.cc:492] T 5a13f36c2b5d48599f2cf5b8a28fb4fe P 0f75989f4187448e90cfb9179cd75ae9: Bootstrap starting.
I20260929 20:27:37.558512 5037 tablet_bootstrap.cc:492] T 5a13f36c2b5d48599f2cf5b8a28fb4fe P 9e75f9ac04d6463b8fc109218fdd82f3: Bootstrap starting.
I20260929 20:27:37.559489 5038 tablet_bootstrap.cc:492] T 5a13f36c2b5d48599f2cf5b8a28fb4fe P d3aeaaf23e884273a2d2fb55a07593c6: Bootstrap starting.
I20260929 20:27:37.559935 5036 tablet_bootstrap.cc:654] T 5a13f36c2b5d48599f2cf5b8a28fb4fe P 0f75989f4187448e90cfb9179cd75ae9: Neither blocks nor log segments found. Creating new log.
I20260929 20:27:37.561352 5036 log.cc:824] T 5a13f36c2b5d48599f2cf5b8a28fb4fe P 0f75989f4187448e90cfb9179cd75ae9: Log is configured to *not* fsync() on all Append() calls
I20260929 20:27:37.562423 5037 tablet_bootstrap.cc:654] T 5a13f36c2b5d48599f2cf5b8a28fb4fe P 9e75f9ac04d6463b8fc109218fdd82f3: Neither blocks nor log segments found. Creating new log.
I20260929 20:27:37.562649 5038 tablet_bootstrap.cc:654] T 5a13f36c2b5d48599f2cf5b8a28fb4fe P d3aeaaf23e884273a2d2fb55a07593c6: Neither blocks nor log segments found. Creating new log.
I20260929 20:27:37.563755 5037 log.cc:824] T 5a13f36c2b5d48599f2cf5b8a28fb4fe P 9e75f9ac04d6463b8fc109218fdd82f3: Log is configured to *not* fsync() on all Append() calls
I20260929 20:27:37.563839 5038 log.cc:824] T 5a13f36c2b5d48599f2cf5b8a28fb4fe P d3aeaaf23e884273a2d2fb55a07593c6: Log is configured to *not* fsync() on all Append() calls
I20260929 20:27:37.564862 5036 tablet_bootstrap.cc:492] T 5a13f36c2b5d48599f2cf5b8a28fb4fe P 0f75989f4187448e90cfb9179cd75ae9: No bootstrap required, opened a new log
I20260929 20:27:37.565155 5036 ts_tablet_manager.cc:1397] T 5a13f36c2b5d48599f2cf5b8a28fb4fe P 0f75989f4187448e90cfb9179cd75ae9: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.003s
I20260929 20:27:37.567010 5037 tablet_bootstrap.cc:492] T 5a13f36c2b5d48599f2cf5b8a28fb4fe P 9e75f9ac04d6463b8fc109218fdd82f3: No bootstrap required, opened a new log
I20260929 20:27:37.567016 5038 tablet_bootstrap.cc:492] T 5a13f36c2b5d48599f2cf5b8a28fb4fe P d3aeaaf23e884273a2d2fb55a07593c6: No bootstrap required, opened a new log
I20260929 20:27:37.567276 5037 ts_tablet_manager.cc:1397] T 5a13f36c2b5d48599f2cf5b8a28fb4fe P 9e75f9ac04d6463b8fc109218fdd82f3: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.002s
I20260929 20:27:37.567270 5038 ts_tablet_manager.cc:1397] T 5a13f36c2b5d48599f2cf5b8a28fb4fe P d3aeaaf23e884273a2d2fb55a07593c6: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20260929 20:27:37.572667 5036 raft_consensus.cc:348] T 5a13f36c2b5d48599f2cf5b8a28fb4fe P 0f75989f4187448e90cfb9179cd75ae9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3aeaaf23e884273a2d2fb55a07593c6" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 44117 } } peers { permanent_uuid: "9e75f9ac04d6463b8fc109218fdd82f3" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 43455 } } peers { permanent_uuid: "0f75989f4187448e90cfb9179cd75ae9" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 46007 } }
I20260929 20:27:37.573050 5036 raft_consensus.cc:374] T 5a13f36c2b5d48599f2cf5b8a28fb4fe P 0f75989f4187448e90cfb9179cd75ae9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:27:37.573164 5036 raft_consensus.cc:729] T 5a13f36c2b5d48599f2cf5b8a28fb4fe P 0f75989f4187448e90cfb9179cd75ae9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f75989f4187448e90cfb9179cd75ae9, State: Initialized, Role: FOLLOWER
I20260929 20:27:37.573770 5036 consensus_queue.cc:260] T 5a13f36c2b5d48599f2cf5b8a28fb4fe P 0f75989f4187448e90cfb9179cd75ae9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3aeaaf23e884273a2d2fb55a07593c6" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 44117 } } peers { permanent_uuid: "9e75f9ac04d6463b8fc109218fdd82f3" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 43455 } } peers { permanent_uuid: "0f75989f4187448e90cfb9179cd75ae9" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 46007 } }
I20260929 20:27:37.575592 5036 ts_tablet_manager.cc:1428] T 5a13f36c2b5d48599f2cf5b8a28fb4fe P 0f75989f4187448e90cfb9179cd75ae9: Time spent starting tablet: real 0.010s user 0.012s sys 0.000s
I20260929 20:27:37.578222 5038 raft_consensus.cc:348] T 5a13f36c2b5d48599f2cf5b8a28fb4fe P d3aeaaf23e884273a2d2fb55a07593c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3aeaaf23e884273a2d2fb55a07593c6" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 44117 } } peers { permanent_uuid: "9e75f9ac04d6463b8fc109218fdd82f3" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 43455 } } peers { permanent_uuid: "0f75989f4187448e90cfb9179cd75ae9" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 46007 } }
I20260929 20:27:37.578222 5037 raft_consensus.cc:348] T 5a13f36c2b5d48599f2cf5b8a28fb4fe P 9e75f9ac04d6463b8fc109218fdd82f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3aeaaf23e884273a2d2fb55a07593c6" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 44117 } } peers { permanent_uuid: "9e75f9ac04d6463b8fc109218fdd82f3" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 43455 } } peers { permanent_uuid: "0f75989f4187448e90cfb9179cd75ae9" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 46007 } }
I20260929 20:27:37.578663 5037 raft_consensus.cc:374] T 5a13f36c2b5d48599f2cf5b8a28fb4fe P 9e75f9ac04d6463b8fc109218fdd82f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:27:37.578657 5038 raft_consensus.cc:374] T 5a13f36c2b5d48599f2cf5b8a28fb4fe P d3aeaaf23e884273a2d2fb55a07593c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:27:37.578752 5037 raft_consensus.cc:729] T 5a13f36c2b5d48599f2cf5b8a28fb4fe P 9e75f9ac04d6463b8fc109218fdd82f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e75f9ac04d6463b8fc109218fdd82f3, State: Initialized, Role: FOLLOWER
I20260929 20:27:37.578752 5038 raft_consensus.cc:729] T 5a13f36c2b5d48599f2cf5b8a28fb4fe P d3aeaaf23e884273a2d2fb55a07593c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3aeaaf23e884273a2d2fb55a07593c6, State: Initialized, Role: FOLLOWER
I20260929 20:27:37.579552 5037 consensus_queue.cc:260] T 5a13f36c2b5d48599f2cf5b8a28fb4fe P 9e75f9ac04d6463b8fc109218fdd82f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3aeaaf23e884273a2d2fb55a07593c6" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 44117 } } peers { permanent_uuid: "9e75f9ac04d6463b8fc109218fdd82f3" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 43455 } } peers { permanent_uuid: "0f75989f4187448e90cfb9179cd75ae9" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 46007 } }
I20260929 20:27:37.579620 5038 consensus_queue.cc:260] T 5a13f36c2b5d48599f2cf5b8a28fb4fe P d3aeaaf23e884273a2d2fb55a07593c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3aeaaf23e884273a2d2fb55a07593c6" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 44117 } } peers { permanent_uuid: "9e75f9ac04d6463b8fc109218fdd82f3" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 43455 } } peers { permanent_uuid: "0f75989f4187448e90cfb9179cd75ae9" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 46007 } }
I20260929 20:27:37.581868 5017 heartbeater.cc:503] Master 127.17.53.126:44651 was elected leader, sending a full tablet report...
I20260929 20:27:37.583603 5038 ts_tablet_manager.cc:1428] T 5a13f36c2b5d48599f2cf5b8a28fb4fe P d3aeaaf23e884273a2d2fb55a07593c6: Time spent starting tablet: real 0.016s user 0.014s sys 0.000s
I20260929 20:27:37.584390 4884 heartbeater.cc:503] Master 127.17.53.126:44651 was elected leader, sending a full tablet report...
I20260929 20:27:37.585209 5037 ts_tablet_manager.cc:1428] T 5a13f36c2b5d48599f2cf5b8a28fb4fe P 9e75f9ac04d6463b8fc109218fdd82f3: Time spent starting tablet: real 0.018s user 0.011s sys 0.005s
W20260929 20:27:37.662782 4751 tablet.cc:2369] T 5a13f36c2b5d48599f2cf5b8a28fb4fe P 0f75989f4187448e90cfb9179cd75ae9: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 20:27:37.675310 4885 tablet.cc:2369] T 5a13f36c2b5d48599f2cf5b8a28fb4fe P 9e75f9ac04d6463b8fc109218fdd82f3: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 20:27:37.704015 5018 tablet.cc:2369] T 5a13f36c2b5d48599f2cf5b8a28fb4fe P d3aeaaf23e884273a2d2fb55a07593c6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 20:27:37.743134 5044 raft_consensus.cc:482] T 5a13f36c2b5d48599f2cf5b8a28fb4fe P d3aeaaf23e884273a2d2fb55a07593c6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 20:27:37.743522 5044 raft_consensus.cc:504] T 5a13f36c2b5d48599f2cf5b8a28fb4fe P d3aeaaf23e884273a2d2fb55a07593c6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3aeaaf23e884273a2d2fb55a07593c6" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 44117 } } peers { permanent_uuid: "9e75f9ac04d6463b8fc109218fdd82f3" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 43455 } } peers { permanent_uuid: "0f75989f4187448e90cfb9179cd75ae9" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 46007 } }
I20260929 20:27:37.745364 5044 leader_election.cc:288] T 5a13f36c2b5d48599f2cf5b8a28fb4fe P d3aeaaf23e884273a2d2fb55a07593c6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9e75f9ac04d6463b8fc109218fdd82f3 (127.17.53.66:43455), 0f75989f4187448e90cfb9179cd75ae9 (127.17.53.65:46007)
I20260929 20:27:37.751689 5020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:27:37.745610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.66:43455 (local address 127.17.53.67:44217)
0929 20:27:37.746011 (+ 401us) negotiation.cc:107] Waiting for socket to connect
0929 20:27:37.746040 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0929 20:27:37.746214 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:27:37.746647 (+ 433us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:27:37.746651 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:27:37.746692 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:27:37.746905 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:27:37.746921 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:27:37.748172 (+ 1251us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:27:37.748178 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:27:37.749627 (+ 1449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:27:37.749639 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:27:37.750939 (+ 1300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:27:37.750945 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:27:37.751149 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:27:37.751189 (+ 40us) client_negotiation.cc:769] Sending connection context
0929 20:27:37.751301 (+ 112us) client_negotiation.cc:240] Negotiation successful
0929 20:27:37.751446 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":204}
I20260929 20:27:37.751843 5033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:27:37.745718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.53.67:44217 (local address 127.17.53.66:43455)
0929 20:27:37.745956 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0929 20:27:37.745963 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:27:37.746304 (+ 341us) server_negotiation.cc:407] Connection header received
0929 20:27:37.746393 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:27:37.746400 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:27:37.746482 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:27:37.746584 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:27:37.747059 (+ 475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:27:37.748042 (+ 983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:27:37.749789 (+ 1747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:27:37.750765 (+ 976us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:27:37.750938 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:27:37.750992 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0929 20:27:37.751399 (+ 407us) server_negotiation.cc:299] Negotiation successful
0929 20:27:37.751547 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260929 20:27:37.752444 5034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:27:37.746002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.53.67:50777 (local address 127.17.53.65:46007)
0929 20:27:37.746219 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0929 20:27:37.746224 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:27:37.746581 (+ 357us) server_negotiation.cc:407] Connection header received
0929 20:27:37.746770 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:27:37.746776 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:27:37.746879 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:27:37.746976 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:27:37.747590 (+ 614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:27:37.748437 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:27:37.750286 (+ 1849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:27:37.751258 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:27:37.751402 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:27:37.751466 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0929 20:27:37.752009 (+ 543us) server_negotiation.cc:299] Negotiation successful
0929 20:27:37.752159 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260929 20:27:37.753029 5045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:27:37.745670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.65:46007 (local address 127.17.53.67:50777)
0929 20:27:37.746481 (+ 811us) negotiation.cc:107] Waiting for socket to connect
0929 20:27:37.746498 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 20:27:37.746620 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:27:37.747048 (+ 428us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:27:37.747052 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:27:37.747078 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:27:37.747437 (+ 359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:27:37.747450 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:27:37.748604 (+ 1154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:27:37.748608 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:27:37.750130 (+ 1522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:27:37.750144 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:27:37.751478 (+ 1334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:27:37.751485 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:27:37.751683 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:27:37.751723 (+ 40us) client_negotiation.cc:769] Sending connection context
0929 20:27:37.752659 (+ 936us) client_negotiation.cc:240] Negotiation successful
0929 20:27:37.752781 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":697,"thread_start_us":452,"threads_started":1}
I20260929 20:27:37.752890 4839 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a13f36c2b5d48599f2cf5b8a28fb4fe" candidate_uuid: "d3aeaaf23e884273a2d2fb55a07593c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e75f9ac04d6463b8fc109218fdd82f3" is_pre_election: true
I20260929 20:27:37.753419 4839 raft_consensus.cc:2454] T 5a13f36c2b5d48599f2cf5b8a28fb4fe P 9e75f9ac04d6463b8fc109218fdd82f3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3aeaaf23e884273a2d2fb55a07593c6 in term 0.
I20260929 20:27:37.754083 4905 leader_election.cc:302] T 5a13f36c2b5d48599f2cf5b8a28fb4fe P d3aeaaf23e884273a2d2fb55a07593c6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9e75f9ac04d6463b8fc109218fdd82f3, d3aeaaf23e884273a2d2fb55a07593c6; no voters:
I20260929 20:27:37.754082 4705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a13f36c2b5d48599f2cf5b8a28fb4fe" candidate_uuid: "d3aeaaf23e884273a2d2fb55a07593c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f75989f4187448e90cfb9179cd75ae9" is_pre_election: true
I20260929 20:27:37.754515 5044 raft_consensus.cc:2781] T 5a13f36c2b5d48599f2cf5b8a28fb4fe P d3aeaaf23e884273a2d2fb55a07593c6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 20:27:37.754691 5044 raft_consensus.cc:482] T 5a13f36c2b5d48599f2cf5b8a28fb4fe P d3aeaaf23e884273a2d2fb55a07593c6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 20:27:37.754705 4705 raft_consensus.cc:2454] T 5a13f36c2b5d48599f2cf5b8a28fb4fe P 0f75989f4187448e90cfb9179cd75ae9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3aeaaf23e884273a2d2fb55a07593c6 in term 0.
I20260929 20:27:37.754846 5044 raft_consensus.cc:3037] T 5a13f36c2b5d48599f2cf5b8a28fb4fe P d3aeaaf23e884273a2d2fb55a07593c6 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:27:37.757580 5044 raft_consensus.cc:504] T 5a13f36c2b5d48599f2cf5b8a28fb4fe P d3aeaaf23e884273a2d2fb55a07593c6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3aeaaf23e884273a2d2fb55a07593c6" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 44117 } } peers { permanent_uuid: "9e75f9ac04d6463b8fc109218fdd82f3" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 43455 } } peers { permanent_uuid: "0f75989f4187448e90cfb9179cd75ae9" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 46007 } }
I20260929 20:27:37.758370 5044 leader_election.cc:288] T 5a13f36c2b5d48599f2cf5b8a28fb4fe P d3aeaaf23e884273a2d2fb55a07593c6 [CANDIDATE]: Term 1 election: Requested vote from peers 9e75f9ac04d6463b8fc109218fdd82f3 (127.17.53.66:43455), 0f75989f4187448e90cfb9179cd75ae9 (127.17.53.65:46007)
I20260929 20:27:37.758772 4839 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a13f36c2b5d48599f2cf5b8a28fb4fe" candidate_uuid: "d3aeaaf23e884273a2d2fb55a07593c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e75f9ac04d6463b8fc109218fdd82f3"
I20260929 20:27:37.758939 4705 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a13f36c2b5d48599f2cf5b8a28fb4fe" candidate_uuid: "d3aeaaf23e884273a2d2fb55a07593c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f75989f4187448e90cfb9179cd75ae9"
I20260929 20:27:37.759089 4839 raft_consensus.cc:3037] T 5a13f36c2b5d48599f2cf5b8a28fb4fe P 9e75f9ac04d6463b8fc109218fdd82f3 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:27:37.759167 4705 raft_consensus.cc:3037] T 5a13f36c2b5d48599f2cf5b8a28fb4fe P 0f75989f4187448e90cfb9179cd75ae9 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:27:37.762557 4839 raft_consensus.cc:2454] T 5a13f36c2b5d48599f2cf5b8a28fb4fe P 9e75f9ac04d6463b8fc109218fdd82f3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3aeaaf23e884273a2d2fb55a07593c6 in term 1.
I20260929 20:27:37.762557 4705 raft_consensus.cc:2454] T 5a13f36c2b5d48599f2cf5b8a28fb4fe P 0f75989f4187448e90cfb9179cd75ae9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3aeaaf23e884273a2d2fb55a07593c6 in term 1.
I20260929 20:27:37.763074 4906 leader_election.cc:302] T 5a13f36c2b5d48599f2cf5b8a28fb4fe P d3aeaaf23e884273a2d2fb55a07593c6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f75989f4187448e90cfb9179cd75ae9, d3aeaaf23e884273a2d2fb55a07593c6; no voters:
I20260929 20:27:37.763422 5044 raft_consensus.cc:2781] T 5a13f36c2b5d48599f2cf5b8a28fb4fe P d3aeaaf23e884273a2d2fb55a07593c6 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:27:37.763983 5044 raft_consensus.cc:686] T 5a13f36c2b5d48599f2cf5b8a28fb4fe P d3aeaaf23e884273a2d2fb55a07593c6 [term 1 LEADER]: Becoming Leader. State: Replica: d3aeaaf23e884273a2d2fb55a07593c6, State: Running, Role: LEADER
I20260929 20:27:37.764542 5044 consensus_queue.cc:237] T 5a13f36c2b5d48599f2cf5b8a28fb4fe P d3aeaaf23e884273a2d2fb55a07593c6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3aeaaf23e884273a2d2fb55a07593c6" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 44117 } } peers { permanent_uuid: "9e75f9ac04d6463b8fc109218fdd82f3" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 43455 } } peers { permanent_uuid: "0f75989f4187448e90cfb9179cd75ae9" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 46007 } }
I20260929 20:27:37.768950 4560 catalog_manager.cc:5495] T 5a13f36c2b5d48599f2cf5b8a28fb4fe P d3aeaaf23e884273a2d2fb55a07593c6 reported cstate change: term changed from 0 to 1, leader changed from <none> to d3aeaaf23e884273a2d2fb55a07593c6 (127.17.53.67). New cstate: current_term: 1 leader_uuid: "d3aeaaf23e884273a2d2fb55a07593c6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3aeaaf23e884273a2d2fb55a07593c6" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 44117 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9e75f9ac04d6463b8fc109218fdd82f3" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 43455 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0f75989f4187448e90cfb9179cd75ae9" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 46007 } health_report { overall_health: UNKNOWN } } }
I20260929 20:27:37.833039 5032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:27:37.827910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59178 (local address 127.17.53.67:44117)
0929 20:27:37.828098 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0929 20:27:37.828102 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:27:37.828122 (+ 20us) server_negotiation.cc:407] Connection header received
0929 20:27:37.828208 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:27:37.828212 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:27:37.828286 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:27:37.828409 (+ 123us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 20:27:37.828754 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:27:37.829477 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:27:37.831014 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:27:37.831399 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:27:37.831497 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:27:37.831962 (+ 465us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 20:27:37.832462 (+ 500us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 20:27:37.832589 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0929 20:27:37.832751 (+ 162us) server_negotiation.cc:299] Negotiation successful
0929 20:27:37.832814 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260929 20:27:37.841418 4705 raft_consensus.cc:1261] T 5a13f36c2b5d48599f2cf5b8a28fb4fe P 0f75989f4187448e90cfb9179cd75ae9 [term 1 FOLLOWER]: Refusing update from remote peer d3aeaaf23e884273a2d2fb55a07593c6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:27:37.841464 4839 raft_consensus.cc:1261] T 5a13f36c2b5d48599f2cf5b8a28fb4fe P 9e75f9ac04d6463b8fc109218fdd82f3 [term 1 FOLLOWER]: Refusing update from remote peer d3aeaaf23e884273a2d2fb55a07593c6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:27:37.842403 5046 consensus_queue.cc:1035] T 5a13f36c2b5d48599f2cf5b8a28fb4fe P d3aeaaf23e884273a2d2fb55a07593c6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f75989f4187448e90cfb9179cd75ae9" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 46007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:27:37.842777 5044 consensus_queue.cc:1035] T 5a13f36c2b5d48599f2cf5b8a28fb4fe P d3aeaaf23e884273a2d2fb55a07593c6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e75f9ac04d6463b8fc109218fdd82f3" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 43455 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:27:37.853926 5049 mvcc.cc:205] Tried to move back new op lower bound from 7334763142508347392 to 7334763142207340544. Current Snapshot: MvccSnapshot[applied={T|T < 7334763142508347392}]
I20260929 20:27:37.855443 5050 mvcc.cc:205] Tried to move back new op lower bound from 7334763142508347392 to 7334763142207340544. Current Snapshot: MvccSnapshot[applied={T|T < 7334763142508347392}]
I20260929 20:27:37.868165 4561 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59900:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 20:27:37.868438 4561 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59900:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 20:27:37.871953 4561 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 79e73df628da45d2bea460ff89bcef8b: Sending DeleteTablet for 3 replicas of tablet 5a13f36c2b5d48599f2cf5b8a28fb4fe
I20260929 20:27:37.873155 4819 tablet_service.cc:1514] Processing DeleteTablet for tablet 5a13f36c2b5d48599f2cf5b8a28fb4fe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:27:37 UTC) from {username='kudu', principal='kudu/127.17.53.126@KRBTEST.COM'} at 127.0.0.1:40598
I20260929 20:27:37.873165 4952 tablet_service.cc:1514] Processing DeleteTablet for tablet 5a13f36c2b5d48599f2cf5b8a28fb4fe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:27:37 UTC) from {username='kudu', principal='kudu/127.17.53.126@KRBTEST.COM'} at 127.0.0.1:59170
I20260929 20:27:37.873155 4685 tablet_service.cc:1514] Processing DeleteTablet for tablet 5a13f36c2b5d48599f2cf5b8a28fb4fe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:27:37 UTC) from {username='kudu', principal='kudu/127.17.53.126@KRBTEST.COM'} at 127.0.0.1:49244
I20260929 20:27:37.875254 17621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu pid 4622
I20260929 20:27:37.889972 5061 tablet_replica.cc:330] T 5a13f36c2b5d48599f2cf5b8a28fb4fe P 0f75989f4187448e90cfb9179cd75ae9: stopping tablet replica
I20260929 20:27:37.890419 5061 raft_consensus.cc:2229] T 5a13f36c2b5d48599f2cf5b8a28fb4fe P 0f75989f4187448e90cfb9179cd75ae9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 20:27:37.891038 5061 raft_consensus.cc:2258] T 5a13f36c2b5d48599f2cf5b8a28fb4fe P 0f75989f4187448e90cfb9179cd75ae9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:27:37.891479 5060 tablet_replica.cc:330] T 5a13f36c2b5d48599f2cf5b8a28fb4fe P 9e75f9ac04d6463b8fc109218fdd82f3: stopping tablet replica
I20260929 20:27:37.891963 5060 raft_consensus.cc:2229] T 5a13f36c2b5d48599f2cf5b8a28fb4fe P 9e75f9ac04d6463b8fc109218fdd82f3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 20:27:37.892122 5059 tablet_replica.cc:330] T 5a13f36c2b5d48599f2cf5b8a28fb4fe P d3aeaaf23e884273a2d2fb55a07593c6: stopping tablet replica
I20260929 20:27:37.892537 5059 raft_consensus.cc:2229] T 5a13f36c2b5d48599f2cf5b8a28fb4fe P d3aeaaf23e884273a2d2fb55a07593c6 [term 1 LEADER]: Raft consensus shutting down.
I20260929 20:27:37.892583 5060 raft_consensus.cc:2258] T 5a13f36c2b5d48599f2cf5b8a28fb4fe P 9e75f9ac04d6463b8fc109218fdd82f3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:27:37.893339 5059 raft_consensus.cc:2258] T 5a13f36c2b5d48599f2cf5b8a28fb4fe P d3aeaaf23e884273a2d2fb55a07593c6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:27:37.893849 5061 ts_tablet_manager.cc:1905] T 5a13f36c2b5d48599f2cf5b8a28fb4fe P 0f75989f4187448e90cfb9179cd75ae9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:27:37.895030 5060 ts_tablet_manager.cc:1905] T 5a13f36c2b5d48599f2cf5b8a28fb4fe P 9e75f9ac04d6463b8fc109218fdd82f3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:27:37.895478 5059 ts_tablet_manager.cc:1905] T 5a13f36c2b5d48599f2cf5b8a28fb4fe P d3aeaaf23e884273a2d2fb55a07593c6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:27:37.901062 5061 ts_tablet_manager.cc:1918] T 5a13f36c2b5d48599f2cf5b8a28fb4fe P 0f75989f4187448e90cfb9179cd75ae9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 20:27:37.901232 5061 log.cc:1196] T 5a13f36c2b5d48599f2cf5b8a28fb4fe P 0f75989f4187448e90cfb9179cd75ae9: Deleting WAL directory at /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790713339161492-17621-0/minicluster-data/ts-0/wal/wals/5a13f36c2b5d48599f2cf5b8a28fb4fe
I20260929 20:27:37.901800 5061 ts_tablet_manager.cc:1939] T 5a13f36c2b5d48599f2cf5b8a28fb4fe P 0f75989f4187448e90cfb9179cd75ae9: Deleting consensus metadata
I20260929 20:27:37.902261 5060 ts_tablet_manager.cc:1918] T 5a13f36c2b5d48599f2cf5b8a28fb4fe P 9e75f9ac04d6463b8fc109218fdd82f3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 20:27:37.902493 5060 log.cc:1196] T 5a13f36c2b5d48599f2cf5b8a28fb4fe P 9e75f9ac04d6463b8fc109218fdd82f3: Deleting WAL directory at /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790713339161492-17621-0/minicluster-data/ts-1/wal/wals/5a13f36c2b5d48599f2cf5b8a28fb4fe
I20260929 20:27:37.902628 5059 ts_tablet_manager.cc:1918] T 5a13f36c2b5d48599f2cf5b8a28fb4fe P d3aeaaf23e884273a2d2fb55a07593c6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 20:27:37.902809 5059 log.cc:1196] T 5a13f36c2b5d48599f2cf5b8a28fb4fe P d3aeaaf23e884273a2d2fb55a07593c6: Deleting WAL directory at /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790713339161492-17621-0/minicluster-data/ts-2/wal/wals/5a13f36c2b5d48599f2cf5b8a28fb4fe
I20260929 20:27:37.903244 5060 ts_tablet_manager.cc:1939] T 5a13f36c2b5d48599f2cf5b8a28fb4fe P 9e75f9ac04d6463b8fc109218fdd82f3: Deleting consensus metadata
I20260929 20:27:37.903364 5059 ts_tablet_manager.cc:1939] T 5a13f36c2b5d48599f2cf5b8a28fb4fe P d3aeaaf23e884273a2d2fb55a07593c6: Deleting consensus metadata
I20260929 20:27:37.904850 4546 catalog_manager.cc:4841] TS 0f75989f4187448e90cfb9179cd75ae9 (127.17.53.65:46007): tablet 5a13f36c2b5d48599f2cf5b8a28fb4fe (table test-table [id=7d1a56cf18d74308a9117153bf5c4c49]) successfully deleted
I20260929 20:27:37.905485 4545 catalog_manager.cc:4841] TS 9e75f9ac04d6463b8fc109218fdd82f3 (127.17.53.66:43455): tablet 5a13f36c2b5d48599f2cf5b8a28fb4fe (table test-table [id=7d1a56cf18d74308a9117153bf5c4c49]) successfully deleted
Sep 29 20:27:37 dist-test-slave-zbnc krb5kdc[4516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713655, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.53.65@KRBTEST.COM
I20260929 20:27:37.908776 4546 catalog_manager.cc:4841] TS d3aeaaf23e884273a2d2fb55a07593c6 (127.17.53.67:44117): tablet 5a13f36c2b5d48599f2cf5b8a28fb4fe (table test-table [id=7d1a56cf18d74308a9117153bf5c4c49]) successfully deleted
I20260929 20:27:37.916600 5034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:27:37.876736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49254 (local address 127.17.53.65:46007)
0929 20:27:37.876967 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0929 20:27:37.876976 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 20:27:37.883428 (+ 6452us) server_negotiation.cc:407] Connection header received
0929 20:27:37.883655 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:27:37.883664 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:27:37.883787 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:27:37.883918 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:27:37.887244 (+ 3326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:27:37.889386 (+ 2142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:27:37.903512 (+ 14126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:27:37.904209 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:27:37.904358 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:27:37.909682 (+ 5324us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:27:37.909699 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:27:37.909714 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:27:37.909765 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:27:37.913080 (+ 3315us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:27:37.914046 (+ 966us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:27:37.914054 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:27:37.914056 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:27:37.914204 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:27:37.914697 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:27:37.914702 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:27:37.914704 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:27:37.915286 (+ 582us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:27:37.915520 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0929 20:27:37.915993 (+ 473us) server_negotiation.cc:299] Negotiation successful
0929 20:27:37.916179 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260929 20:27:37.917407 4745 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:27:38.015141 17621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu with pid 4622
I20260929 20:27:38.033141 17621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu pid 4756
Sep 29 20:27:38 dist-test-slave-zbnc krb5kdc[4516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713655, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.53.66@KRBTEST.COM
I20260929 20:27:38.050980 5033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:27:38.034047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40600 (local address 127.17.53.66:43455)
0929 20:27:38.034200 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0929 20:27:38.034205 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:27:38.034228 (+ 23us) server_negotiation.cc:407] Connection header received
0929 20:27:38.034446 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:27:38.034450 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:27:38.034523 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:27:38.034606 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:27:38.036267 (+ 1661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:27:38.037035 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:27:38.038919 (+ 1884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:27:38.039331 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:27:38.039448 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:27:38.044658 (+ 5210us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:27:38.044674 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:27:38.044691 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:27:38.044735 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:27:38.048063 (+ 3328us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:27:38.048793 (+ 730us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:27:38.048808 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:27:38.048810 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:27:38.048928 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:27:38.049305 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:27:38.049310 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:27:38.049313 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:27:38.049701 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:27:38.050002 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0929 20:27:38.050355 (+ 353us) server_negotiation.cc:299] Negotiation successful
0929 20:27:38.050578 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260929 20:27:38.051640 4879 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:27:38.146274 17621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu with pid 4756
I20260929 20:27:38.171265 17621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu pid 4889
Sep 29 20:27:38 dist-test-slave-zbnc krb5kdc[4516](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713655, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.53.67@KRBTEST.COM
I20260929 20:27:38.190052 5032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:27:38.172127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59194 (local address 127.17.53.67:44117)
0929 20:27:38.172335 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0929 20:27:38.172341 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:27:38.172363 (+ 22us) server_negotiation.cc:407] Connection header received
0929 20:27:38.172457 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:27:38.172463 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:27:38.172579 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:27:38.172692 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:27:38.174039 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:27:38.175372 (+ 1333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:27:38.177073 (+ 1701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:27:38.177758 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:27:38.177906 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:27:38.182383 (+ 4477us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:27:38.182399 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:27:38.182419 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:27:38.182469 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:27:38.186930 (+ 4461us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:27:38.187638 (+ 708us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:27:38.187648 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:27:38.187651 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:27:38.187874 (+ 223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:27:38.188250 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:27:38.188281 (+ 31us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:27:38.188285 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:27:38.188772 (+ 487us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:27:38.189115 (+ 343us) server_negotiation.cc:1035] Waiting for connection context
0929 20:27:38.189368 (+ 253us) server_negotiation.cc:299] Negotiation successful
0929 20:27:38.189610 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260929 20:27:38.191001 5012 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:27:38.316247 17621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu with pid 4889
I20260929 20:27:38.336555 17621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu pid 4529
I20260929 20:27:38.337658 4591 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:27:38.450662 17621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu with pid 4529
2026-09-29T20:27:38Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (4444 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-29T20:27:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T20:27:38Z Disabled control of system clock
I20260929 20:27:38.528842 17621 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
/tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790713339161492-17621-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790713339161492-17621-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790713339161492-17621-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790713339161492-17621-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.53.126:34863
--webserver_interface=127.17.53.126
--webserver_port=0
--builtin_ntp_servers=127.17.53.84:39811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.53.126:34863
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 20:27:38.865221 5070 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:27:38.865545 5070 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:27:38.865628 5070 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:27:38.873054 5070 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:27:38.873164 5070 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:27:38.873202 5070 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:27:38.873267 5070 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:27:38.873311 5070 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:27:38.882642 5070 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.53.84:39811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790713339161492-17621-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790713339161492-17621-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.53.126:34863
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.53.126:34863
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790713339161492-17621-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.53.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790713339161492-17621-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:27:38.883970 5070 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:27:38.885769 5070 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:27:38.898089 5078 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:27:38.898068 5075 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:27:38.898619 5076 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:27:38.898907 5070 server_base.cc:1034] running on GCE node
I20260929 20:27:38.899708 5070 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:27:38.901221 5070 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:27:38.902468 5070 hybrid_clock.cc:648] HybridClock initialized: now 1790713658902410 us; error 46 us; skew 500 ppm
I20260929 20:27:38.902961 5070 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 20:27:38.905381 5070 webserver.cc:460] Webserver started at http://127.17.53.126:43221/ using document root <none> and password file <none>
I20260929 20:27:38.906091 5070 fs_manager.cc:359] Metadata directory not provided
I20260929 20:27:38.906217 5070 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:27:38.906546 5070 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:27:38.909513 5070 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790713339161492-17621-0/minicluster-data/master-0/data/instance:
uuid: "3f1b9c4d140c4dcd805e1b439f5490f5"
format_stamp: "Formatted at 2026-09-29 20:27:38 on dist-test-slave-zbnc"
server_key: "7a9986be56b484d55945703a0ed76a26"
server_key_iv: "11cd77ad8b5ea01bb204b30aebb9df40"
server_key_version: "encryptionkey@0"
I20260929 20:27:38.910317 5070 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790713339161492-17621-0/minicluster-data/master-0/wal/instance:
uuid: "3f1b9c4d140c4dcd805e1b439f5490f5"
format_stamp: "Formatted at 2026-09-29 20:27:38 on dist-test-slave-zbnc"
server_key: "7a9986be56b484d55945703a0ed76a26"
server_key_iv: "11cd77ad8b5ea01bb204b30aebb9df40"
server_key_version: "encryptionkey@0"
I20260929 20:27:38.915817 5070 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260929 20:27:38.919502 5084 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:27:38.920814 5070 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 20:27:38.921010 5070 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790713339161492-17621-0/minicluster-data/master-0/data,/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790713339161492-17621-0/minicluster-data/master-0/wal
uuid: "3f1b9c4d140c4dcd805e1b439f5490f5"
format_stamp: "Formatted at 2026-09-29 20:27:38 on dist-test-slave-zbnc"
server_key: "7a9986be56b484d55945703a0ed76a26"
server_key_iv: "11cd77ad8b5ea01bb204b30aebb9df40"
server_key_version: "encryptionkey@0"
I20260929 20:27:38.921223 5070 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790713339161492-17621-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790713339161492-17621-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790713339161492-17621-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:27:38.952314 5070 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:27:38.953272 5070 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:27:38.953565 5070 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:27:38.972350 5070 rpc_server.cc:307] RPC server started. Bound to: 127.17.53.126:34863
I20260929 20:27:38.972374 5135 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.53.126:34863 every 8 connection(s)
I20260929 20:27:38.973814 5070 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790713339161492-17621-0/minicluster-data/master-0/data/info.pb
I20260929 20:27:38.977624 5136 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:27:38.983484 17621 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu as pid 5070
I20260929 20:27:38.983661 17621 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790713339161492-17621-0/minicluster-data/master-0/wal/instance
I20260929 20:27:38.984090 17621 external_mini_cluster.cc:1428] Setting key 50b3ac947c9eaeff736f5a1024fd400c
I20260929 20:27:38.988719 5136 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f1b9c4d140c4dcd805e1b439f5490f5: Bootstrap starting.
I20260929 20:27:38.991815 5136 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3f1b9c4d140c4dcd805e1b439f5490f5: Neither blocks nor log segments found. Creating new log.
I20260929 20:27:38.992805 5136 log.cc:824] T 00000000000000000000000000000000 P 3f1b9c4d140c4dcd805e1b439f5490f5: Log is configured to *not* fsync() on all Append() calls
I20260929 20:27:38.995618 5136 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f1b9c4d140c4dcd805e1b439f5490f5: No bootstrap required, opened a new log
I20260929 20:27:38.997546 5139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:27:38.986051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42252 (local address 127.17.53.126:34863)
0929 20:27:38.987100 (+ 1049us) server_negotiation.cc:206] Beginning negotiation
0929 20:27:38.987111 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 20:27:38.987159 (+ 48us) server_negotiation.cc:407] Connection header received
0929 20:27:38.988869 (+ 1710us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:27:38.988907 (+ 38us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:27:38.989375 (+ 468us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:27:38.989762 (+ 387us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:27:38.990351 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:27:38.991676 (+ 1325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:27:38.993132 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:27:38.993921 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:27:38.994118 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:27:38.995255 (+ 1137us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:27:38.995262 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:27:38.995277 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:27:38.995836 (+ 559us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:27:38.995913 (+ 77us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:27:38.995934 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:27:38.996063 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0929 20:27:38.996271 (+ 208us) server_negotiation.cc:299] Negotiation successful
0929 20:27:38.996569 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":561,"thread_start_us":330,"threads_started":1}
I20260929 20:27:39.003778 5136 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3f1b9c4d140c4dcd805e1b439f5490f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f1b9c4d140c4dcd805e1b439f5490f5" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 34863 } }
I20260929 20:27:39.004164 5136 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3f1b9c4d140c4dcd805e1b439f5490f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:27:39.004264 5136 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3f1b9c4d140c4dcd805e1b439f5490f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f1b9c4d140c4dcd805e1b439f5490f5, State: Initialized, Role: FOLLOWER
I20260929 20:27:39.004896 5136 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3f1b9c4d140c4dcd805e1b439f5490f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f1b9c4d140c4dcd805e1b439f5490f5" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 34863 } }
I20260929 20:27:39.005177 5136 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3f1b9c4d140c4dcd805e1b439f5490f5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 20:27:39.005338 5136 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3f1b9c4d140c4dcd805e1b439f5490f5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 20:27:39.005535 5136 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3f1b9c4d140c4dcd805e1b439f5490f5 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:27:39.007795 5136 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3f1b9c4d140c4dcd805e1b439f5490f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f1b9c4d140c4dcd805e1b439f5490f5" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 34863 } }
I20260929 20:27:39.008292 5136 leader_election.cc:302] T 00000000000000000000000000000000 P 3f1b9c4d140c4dcd805e1b439f5490f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3f1b9c4d140c4dcd805e1b439f5490f5; no voters:
I20260929 20:27:39.008831 5136 leader_election.cc:288] T 00000000000000000000000000000000 P 3f1b9c4d140c4dcd805e1b439f5490f5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 20:27:39.008972 5141 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3f1b9c4d140c4dcd805e1b439f5490f5 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:27:39.009723 5141 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3f1b9c4d140c4dcd805e1b439f5490f5 [term 1 LEADER]: Becoming Leader. State: Replica: 3f1b9c4d140c4dcd805e1b439f5490f5, State: Running, Role: LEADER
I20260929 20:27:39.010213 5141 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3f1b9c4d140c4dcd805e1b439f5490f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f1b9c4d140c4dcd805e1b439f5490f5" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 34863 } }
I20260929 20:27:39.011013 5136 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3f1b9c4d140c4dcd805e1b439f5490f5 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:27:39.013541 5142 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f1b9c4d140c4dcd805e1b439f5490f5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3f1b9c4d140c4dcd805e1b439f5490f5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f1b9c4d140c4dcd805e1b439f5490f5" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 34863 } } }
I20260929 20:27:39.013450 5143 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f1b9c4d140c4dcd805e1b439f5490f5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3f1b9c4d140c4dcd805e1b439f5490f5. Latest consensus state: current_term: 1 leader_uuid: "3f1b9c4d140c4dcd805e1b439f5490f5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3f1b9c4d140c4dcd805e1b439f5490f5" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 34863 } } }
I20260929 20:27:39.013983 5142 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f1b9c4d140c4dcd805e1b439f5490f5 [sys.catalog]: This master's current role is: LEADER
I20260929 20:27:39.013979 5143 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f1b9c4d140c4dcd805e1b439f5490f5 [sys.catalog]: This master's current role is: LEADER
I20260929 20:27:39.019761 5148 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:27:39.025523 5148 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:27:39.034152 5148 catalog_manager.cc:1347] Generated new cluster ID: a324090ff70648428c9f4f10a14bbcd2
I20260929 20:27:39.034268 5148 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:27:39.056977 5148 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 20:27:39.059149 5148 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:27:39.073402 5148 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3f1b9c4d140c4dcd805e1b439f5490f5: Generated new TSK 0
I20260929 20:27:39.074282 5148 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 20:27:39.086127 17621 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
/tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790713339161492-17621-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790713339161492-17621-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790713339161492-17621-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790713339161492-17621-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.53.65:0
--local_ip_for_outbound_sockets=127.17.53.65
--webserver_interface=127.17.53.65
--webserver_port=0
--tserver_master_addrs=127.17.53.126:34863
--builtin_ntp_servers=127.17.53.84:39811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260929 20:27:39.426322 5160 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:27:39.426607 5160 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:27:39.426769 5160 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:27:39.434043 5160 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:27:39.434141 5160 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:27:39.434310 5160 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.53.65
I20260929 20:27:39.443600 5160 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.53.84:39811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790713339161492-17621-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790713339161492-17621-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.53.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790713339161492-17621-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.17.53.65
--webserver_port=0
--tserver_master_addrs=127.17.53.126:34863
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.53.65
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790713339161492-17621-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:27:39.444902 5160 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:27:39.446730 5160 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:27:39.459825 5165 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:27:39.459862 5168 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:27:39.459947 5166 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:27:39.460649 5160 server_base.cc:1034] running on GCE node
I20260929 20:27:39.461264 5160 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:27:39.462203 5160 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:27:39.463465 5160 hybrid_clock.cc:648] HybridClock initialized: now 1790713659463414 us; error 44 us; skew 500 ppm
I20260929 20:27:39.463796 5160 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 20:27:39.466069 5160 webserver.cc:460] Webserver started at http://127.17.53.65:40309/ using document root <none> and password file <none>
I20260929 20:27:39.466754 5160 fs_manager.cc:359] Metadata directory not provided
I20260929 20:27:39.466890 5160 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:27:39.467203 5160 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:27:39.469992 5160 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790713339161492-17621-0/minicluster-data/ts-0/data/instance:
uuid: "ba2bfffde3d04a96b1fbe92992010591"
format_stamp: "Formatted at 2026-09-29 20:27:39 on dist-test-slave-zbnc"
server_key: "8e81bd3713163c0f2d9bd77669fae390"
server_key_iv: "d518c323f5a056d66f91cd86851a1b02"
server_key_version: "encryptionkey@0"
I20260929 20:27:39.470769 5160 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790713339161492-17621-0/minicluster-data/ts-0/wal/instance:
uuid: "ba2bfffde3d04a96b1fbe92992010591"
format_stamp: "Formatted at 2026-09-29 20:27:39 on dist-test-slave-zbnc"
server_key: "8e81bd3713163c0f2d9bd77669fae390"
server_key_iv: "d518c323f5a056d66f91cd86851a1b02"
server_key_version: "encryptionkey@0"
I20260929 20:27:39.476377 5160 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260929 20:27:39.480244 5174 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:27:39.481705 5160 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 20:27:39.481956 5160 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790713339161492-17621-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790713339161492-17621-0/minicluster-data/ts-0/wal
uuid: "ba2bfffde3d04a96b1fbe92992010591"
format_stamp: "Formatted at 2026-09-29 20:27:39 on dist-test-slave-zbnc"
server_key: "8e81bd3713163c0f2d9bd77669fae390"
server_key_iv: "d518c323f5a056d66f91cd86851a1b02"
server_key_version: "encryptionkey@0"
I20260929 20:27:39.482177 5160 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790713339161492-17621-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790713339161492-17621-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790713339161492-17621-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:27:39.510764 5160 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:27:39.511861 5160 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:27:39.512226 5160 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:27:39.513356 5160 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:27:39.515090 5160 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:27:39.515224 5160 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:27:39.515365 5160 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:27:39.515458 5160 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:27:39.551975 5160 rpc_server.cc:307] RPC server started. Bound to: 127.17.53.65:35343
I20260929 20:27:39.552062 5286 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.53.65:35343 every 8 connection(s)
I20260929 20:27:39.553341 5160 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790713339161492-17621-0/minicluster-data/ts-0/data/info.pb
I20260929 20:27:39.561951 17621 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu as pid 5160
I20260929 20:27:39.562182 17621 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790713339161492-17621-0/minicluster-data/ts-0/wal/instance
I20260929 20:27:39.562825 17621 external_mini_cluster.cc:1428] Setting key a4ab971d393c162507b1fd5c43d0c9ba
I20260929 20:27:39.566028 17621 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
/tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790713339161492-17621-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790713339161492-17621-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790713339161492-17621-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790713339161492-17621-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.53.66:0
--local_ip_for_outbound_sockets=127.17.53.66
--webserver_interface=127.17.53.66
--webserver_port=0
--tserver_master_addrs=127.17.53.126:34863
--builtin_ntp_servers=127.17.53.84:39811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 20:27:39.567175 5289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:27:39.556062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.53.65:48291 (local address 127.17.53.126:34863)
0929 20:27:39.556529 (+ 467us) server_negotiation.cc:206] Beginning negotiation
0929 20:27:39.556536 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:27:39.557965 (+ 1429us) server_negotiation.cc:407] Connection header received
0929 20:27:39.559853 (+ 1888us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:27:39.559858 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:27:39.559927 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:27:39.560030 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:27:39.561131 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:27:39.561936 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:27:39.564215 (+ 2279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:27:39.564699 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:27:39.564820 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:27:39.566261 (+ 1441us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:27:39.566266 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:27:39.566270 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:27:39.566291 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:27:39.566340 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:27:39.566356 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:27:39.566462 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0929 20:27:39.566684 (+ 222us) server_negotiation.cc:299] Negotiation successful
0929 20:27:39.566785 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":342,"thread_start_us":223,"threads_started":1}
I20260929 20:27:39.568230 5290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:27:39.556702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:34863 (local address 127.17.53.65:48291)
0929 20:27:39.557657 (+ 955us) negotiation.cc:107] Waiting for socket to connect
0929 20:27:39.557696 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0929 20:27:39.559570 (+ 1874us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:27:39.560256 (+ 686us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:27:39.560264 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:27:39.560352 (+ 88us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:27:39.560968 (+ 616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:27:39.560986 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:27:39.562114 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:27:39.562121 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:27:39.564081 (+ 1960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:27:39.564096 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:27:39.564945 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:27:39.564955 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:27:39.565208 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:27:39.565946 (+ 738us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 20:27:39.565949 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:27:39.565993 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 20:27:39.566068 (+ 75us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:27:39.566459 (+ 391us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:27:39.566471 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:27:39.566481 (+ 10us) client_negotiation.cc:769] Sending connection context
0929 20:27:39.566674 (+ 193us) client_negotiation.cc:240] Negotiation successful
0929 20:27:39.567190 (+ 516us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":506,"thread_start_us":301,"threads_started":1}
I20260929 20:27:39.569969 5287 heartbeater.cc:347] Connected to a master server at 127.17.53.126:34863
I20260929 20:27:39.570286 5287 heartbeater.cc:464] Registering TS with master...
I20260929 20:27:39.571089 5287 heartbeater.cc:511] Master 127.17.53.126:34863 requested a full tablet report, sending...
I20260929 20:27:39.573506 5101 ts_manager.cc:194] Registered new tserver with Master: ba2bfffde3d04a96b1fbe92992010591 (127.17.53.65:35343)
I20260929 20:27:39.575186 5101 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.17.53.65:48291
W20260929 20:27:39.895680 5291 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:27:39.895953 5291 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:27:39.896118 5291 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:27:39.903847 5291 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:27:39.904003 5291 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:27:39.904168 5291 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.53.66
I20260929 20:27:39.913681 5291 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.53.84:39811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790713339161492-17621-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790713339161492-17621-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.53.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790713339161492-17621-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.17.53.66
--webserver_port=0
--tserver_master_addrs=127.17.53.126:34863
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.53.66
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790713339161492-17621-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:27:39.915134 5291 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:27:39.916951 5291 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:27:39.930558 5296 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:27:39.930522 5299 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:27:39.930599 5297 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:27:39.931928 5291 server_base.cc:1034] running on GCE node
I20260929 20:27:39.932567 5291 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:27:39.933495 5291 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:27:39.934720 5291 hybrid_clock.cc:648] HybridClock initialized: now 1790713659934665 us; error 60 us; skew 500 ppm
I20260929 20:27:39.935079 5291 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 20:27:39.937352 5291 webserver.cc:460] Webserver started at http://127.17.53.66:42113/ using document root <none> and password file <none>
I20260929 20:27:39.938062 5291 fs_manager.cc:359] Metadata directory not provided
I20260929 20:27:39.938191 5291 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:27:39.938509 5291 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:27:39.941448 5291 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790713339161492-17621-0/minicluster-data/ts-1/data/instance:
uuid: "2393d6f0b3ca400889cd0973fae0793f"
format_stamp: "Formatted at 2026-09-29 20:27:39 on dist-test-slave-zbnc"
server_key: "2a70a68ad0fcf060d60ac97a32339137"
server_key_iv: "2d71c837698b275358b72dfb116422cf"
server_key_version: "encryptionkey@0"
I20260929 20:27:39.942268 5291 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790713339161492-17621-0/minicluster-data/ts-1/wal/instance:
uuid: "2393d6f0b3ca400889cd0973fae0793f"
format_stamp: "Formatted at 2026-09-29 20:27:39 on dist-test-slave-zbnc"
server_key: "2a70a68ad0fcf060d60ac97a32339137"
server_key_iv: "2d71c837698b275358b72dfb116422cf"
server_key_version: "encryptionkey@0"
I20260929 20:27:39.947453 5291 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.007s
I20260929 20:27:39.950755 5305 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:27:39.952114 5291 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 20:27:39.952289 5291 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790713339161492-17621-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790713339161492-17621-0/minicluster-data/ts-1/wal
uuid: "2393d6f0b3ca400889cd0973fae0793f"
format_stamp: "Formatted at 2026-09-29 20:27:39 on dist-test-slave-zbnc"
server_key: "2a70a68ad0fcf060d60ac97a32339137"
server_key_iv: "2d71c837698b275358b72dfb116422cf"
server_key_version: "encryptionkey@0"
I20260929 20:27:39.952526 5291 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790713339161492-17621-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790713339161492-17621-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790713339161492-17621-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:27:39.984905 5291 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:27:39.985827 5291 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:27:39.986130 5291 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:27:39.987165 5291 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:27:39.988637 5291 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:27:39.988762 5291 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:27:39.988880 5291 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:27:39.988978 5291 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:27:40.021733 5291 rpc_server.cc:307] RPC server started. Bound to: 127.17.53.66:33623
I20260929 20:27:40.021821 5417 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.53.66:33623 every 8 connection(s)
I20260929 20:27:40.023154 5291 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790713339161492-17621-0/minicluster-data/ts-1/data/info.pb
I20260929 20:27:40.033326 17621 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu as pid 5291
I20260929 20:27:40.033515 17621 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790713339161492-17621-0/minicluster-data/ts-1/wal/instance
I20260929 20:27:40.034127 17621 external_mini_cluster.cc:1428] Setting key 005a8ca0fad6da4afc20e3501819bb1d
I20260929 20:27:40.036273 5289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:27:40.025791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.53.66:49149 (local address 127.17.53.126:34863)
0929 20:27:40.025996 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0929 20:27:40.026000 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:27:40.027712 (+ 1712us) server_negotiation.cc:407] Connection header received
0929 20:27:40.029530 (+ 1818us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:27:40.029535 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:27:40.029603 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:27:40.029700 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:27:40.030756 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:27:40.031521 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:27:40.033135 (+ 1614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:27:40.033776 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:27:40.033909 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:27:40.035331 (+ 1422us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:27:40.035336 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:27:40.035339 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:27:40.035358 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:27:40.035406 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:27:40.035419 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:27:40.035549 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0929 20:27:40.035841 (+ 292us) server_negotiation.cc:299] Negotiation successful
0929 20:27:40.035944 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260929 20:27:40.037328 5420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:27:40.026442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:34863 (local address 127.17.53.66:49149)
0929 20:27:40.027537 (+ 1095us) negotiation.cc:107] Waiting for socket to connect
0929 20:27:40.027586 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0929 20:27:40.029280 (+ 1694us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:27:40.029912 (+ 632us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:27:40.029921 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:27:40.030038 (+ 117us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:27:40.030603 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:27:40.030618 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:27:40.031644 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:27:40.031648 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:27:40.032927 (+ 1279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:27:40.032943 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:27:40.033997 (+ 1054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:27:40.034002 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:27:40.034244 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:27:40.034999 (+ 755us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 20:27:40.035001 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:27:40.035042 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 20:27:40.035122 (+ 80us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:27:40.035550 (+ 428us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:27:40.035566 (+ 16us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:27:40.035575 (+ 9us) client_negotiation.cc:769] Sending connection context
0929 20:27:40.035832 (+ 257us) client_negotiation.cc:240] Negotiation successful
0929 20:27:40.036273 (+ 441us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":549,"thread_start_us":353,"threads_started":1}
I20260929 20:27:40.037851 17621 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
/tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790713339161492-17621-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790713339161492-17621-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790713339161492-17621-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790713339161492-17621-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.17.53.67:0
--local_ip_for_outbound_sockets=127.17.53.67
--webserver_interface=127.17.53.67
--webserver_port=0
--tserver_master_addrs=127.17.53.126:34863
--builtin_ntp_servers=127.17.53.84:39811
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260929 20:27:40.039096 5418 heartbeater.cc:347] Connected to a master server at 127.17.53.126:34863
I20260929 20:27:40.039399 5418 heartbeater.cc:464] Registering TS with master...
I20260929 20:27:40.040159 5418 heartbeater.cc:511] Master 127.17.53.126:34863 requested a full tablet report, sending...
I20260929 20:27:40.041565 5101 ts_manager.cc:194] Registered new tserver with Master: 2393d6f0b3ca400889cd0973fae0793f (127.17.53.66:33623)
I20260929 20:27:40.042374 5101 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.17.53.66:49149
W20260929 20:27:40.377780 5421 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:27:40.378093 5421 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:27:40.378254 5421 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:27:40.385696 5421 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:27:40.385834 5421 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:27:40.386011 5421 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.17.53.67
I20260929 20:27:40.395043 5421 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.53.84:39811
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790713339161492-17621-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790713339161492-17621-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.17.53.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790713339161492-17621-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.17.53.67
--webserver_port=0
--tserver_master_addrs=127.17.53.126:34863
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.17.53.67
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790713339161492-17621-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:27:40.396306 5421 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:27:40.398059 5421 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:27:40.410848 5426 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:27:40.410887 5427 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:27:40.411029 5429 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:27:40.411924 5421 server_base.cc:1034] running on GCE node
I20260929 20:27:40.412465 5421 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:27:40.413367 5421 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:27:40.414569 5421 hybrid_clock.cc:648] HybridClock initialized: now 1790713660414530 us; error 37 us; skew 500 ppm
I20260929 20:27:40.414973 5421 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 20:27:40.417184 5421 webserver.cc:460] Webserver started at http://127.17.53.67:34425/ using document root <none> and password file <none>
I20260929 20:27:40.417918 5421 fs_manager.cc:359] Metadata directory not provided
I20260929 20:27:40.418037 5421 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:27:40.418354 5421 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:27:40.421309 5421 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790713339161492-17621-0/minicluster-data/ts-2/data/instance:
uuid: "cc2b2cf56f104dad862ab3d71d0c2409"
format_stamp: "Formatted at 2026-09-29 20:27:40 on dist-test-slave-zbnc"
server_key: "83c2138c58787ac442016acb109f6606"
server_key_iv: "aaeadd5641e1c81843a4fa331aa25580"
server_key_version: "encryptionkey@0"
I20260929 20:27:40.422098 5421 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790713339161492-17621-0/minicluster-data/ts-2/wal/instance:
uuid: "cc2b2cf56f104dad862ab3d71d0c2409"
format_stamp: "Formatted at 2026-09-29 20:27:40 on dist-test-slave-zbnc"
server_key: "83c2138c58787ac442016acb109f6606"
server_key_iv: "aaeadd5641e1c81843a4fa331aa25580"
server_key_version: "encryptionkey@0"
I20260929 20:27:40.427591 5421 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260929 20:27:40.431162 5435 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:27:40.432552 5421 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 20:27:40.432734 5421 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790713339161492-17621-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790713339161492-17621-0/minicluster-data/ts-2/wal
uuid: "cc2b2cf56f104dad862ab3d71d0c2409"
format_stamp: "Formatted at 2026-09-29 20:27:40 on dist-test-slave-zbnc"
server_key: "83c2138c58787ac442016acb109f6606"
server_key_iv: "aaeadd5641e1c81843a4fa331aa25580"
server_key_version: "encryptionkey@0"
I20260929 20:27:40.432960 5421 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790713339161492-17621-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790713339161492-17621-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790713339161492-17621-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:27:40.472030 5421 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:27:40.472993 5421 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:27:40.473308 5421 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:27:40.474332 5421 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260929 20:27:40.476043 5421 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 20:27:40.476167 5421 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 20:27:40.476286 5421 ts_tablet_manager.cc:610] Registered 0 tablets
I20260929 20:27:40.476375 5421 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 20:27:40.511114 5421 rpc_server.cc:307] RPC server started. Bound to: 127.17.53.67:40291
I20260929 20:27:40.511202 5547 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.53.67:40291 every 8 connection(s)
I20260929 20:27:40.512578 5421 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790713339161492-17621-0/minicluster-data/ts-2/data/info.pb
I20260929 20:27:40.516891 17621 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu as pid 5421
I20260929 20:27:40.517081 17621 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790713339161492-17621-0/minicluster-data/ts-2/wal/instance
I20260929 20:27:40.517685 17621 external_mini_cluster.cc:1428] Setting key a9e839a6725250ee682b40e13ab54c2c
I20260929 20:27:40.526337 5289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:27:40.515402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.53.67:43003 (local address 127.17.53.126:34863)
0929 20:27:40.515590 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0929 20:27:40.515595 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:27:40.517195 (+ 1600us) server_negotiation.cc:407] Connection header received
0929 20:27:40.519721 (+ 2526us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:27:40.519726 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:27:40.519881 (+ 155us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:27:40.519986 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:27:40.521288 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:27:40.522047 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:27:40.523653 (+ 1606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:27:40.524076 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:27:40.524170 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:27:40.525588 (+ 1418us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:27:40.525592 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:27:40.525595 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:27:40.525614 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:27:40.525642 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:27:40.525654 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:27:40.525762 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0929 20:27:40.525980 (+ 218us) server_negotiation.cc:299] Negotiation successful
0929 20:27:40.526081 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260929 20:27:40.527400 5550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:27:40.515944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:34863 (local address 127.17.53.67:43003)
0929 20:27:40.517002 (+ 1058us) negotiation.cc:107] Waiting for socket to connect
0929 20:27:40.517061 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0929 20:27:40.519372 (+ 2311us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:27:40.520247 (+ 875us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:27:40.520255 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:27:40.520349 (+ 94us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:27:40.521123 (+ 774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:27:40.521143 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:27:40.522182 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:27:40.522186 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:27:40.523524 (+ 1338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:27:40.523536 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:27:40.524255 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:27:40.524259 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:27:40.524456 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:27:40.525296 (+ 840us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0929 20:27:40.525299 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:27:40.525338 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0929 20:27:40.525422 (+ 84us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:27:40.525784 (+ 362us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:27:40.525794 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:27:40.525802 (+ 8us) client_negotiation.cc:769] Sending connection context
0929 20:27:40.525955 (+ 153us) client_negotiation.cc:240] Negotiation successful
0929 20:27:40.526288 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":433,"thread_start_us":256,"threads_started":1}
I20260929 20:27:40.528776 5548 heartbeater.cc:347] Connected to a master server at 127.17.53.126:34863
I20260929 20:27:40.529072 5548 heartbeater.cc:464] Registering TS with master...
I20260929 20:27:40.529821 5548 heartbeater.cc:511] Master 127.17.53.126:34863 requested a full tablet report, sending...
I20260929 20:27:40.531410 5101 ts_manager.cc:194] Registered new tserver with Master: cc2b2cf56f104dad862ab3d71d0c2409 (127.17.53.67:40291)
I20260929 20:27:40.532519 5101 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.17.53.67:43003
I20260929 20:27:40.533278 17621 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260929 20:27:40.544428 5289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:27:40.537520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59580 (local address 127.17.53.126:34863)
0929 20:27:40.537758 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0929 20:27:40.537763 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:27:40.539339 (+ 1576us) server_negotiation.cc:407] Connection header received
0929 20:27:40.539474 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:27:40.539478 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:27:40.539553 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:27:40.539638 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:27:40.540079 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:27:40.540752 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:27:40.541988 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:27:40.542422 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:27:40.542515 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:27:40.543707 (+ 1192us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:27:40.543713 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:27:40.543716 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:27:40.543745 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:27:40.543779 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:27:40.543792 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:27:40.543871 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0929 20:27:40.544019 (+ 148us) server_negotiation.cc:299] Negotiation successful
0929 20:27:40.544094 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260929 20:27:40.548617 5100 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:59580:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260929 20:27:40.557840 5100 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 20:27:40.578938 5287 heartbeater.cc:503] Master 127.17.53.126:34863 was elected leader, sending a full tablet report...
I20260929 20:27:40.584335 5562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:27:40.570847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.66:33623 (local address 127.0.0.1:35964)
0929 20:27:40.572619 (+ 1772us) negotiation.cc:107] Waiting for socket to connect
0929 20:27:40.572642 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0929 20:27:40.572788 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:27:40.574006 (+ 1218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:27:40.574017 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:27:40.574067 (+ 50us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:27:40.574537 (+ 470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:27:40.574553 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:27:40.577559 (+ 3006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:27:40.577564 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:27:40.581111 (+ 3547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:27:40.581131 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:27:40.583376 (+ 2245us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:27:40.583382 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:27:40.583611 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:27:40.583669 (+ 58us) client_negotiation.cc:769] Sending connection context
0929 20:27:40.583793 (+ 124us) client_negotiation.cc:240] Negotiation successful
0929 20:27:40.583993 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1569,"thread_start_us":1396,"threads_started":1}
I20260929 20:27:40.585646 5563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:27:40.571399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35964 (local address 127.17.53.66:33623)
0929 20:27:40.572097 (+ 698us) server_negotiation.cc:206] Beginning negotiation
0929 20:27:40.572110 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 20:27:40.572961 (+ 851us) server_negotiation.cc:407] Connection header received
0929 20:27:40.573090 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:27:40.573101 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:27:40.573453 (+ 352us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:27:40.573677 (+ 224us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:27:40.574820 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:27:40.576970 (+ 2150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:27:40.581354 (+ 4384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:27:40.582826 (+ 1472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:27:40.582976 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:27:40.583255 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0929 20:27:40.584711 (+ 1456us) server_negotiation.cc:299] Negotiation successful
0929 20:27:40.585288 (+ 577us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":478,"thread_start_us":365,"threads_started":1}
I20260929 20:27:40.585806 5560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:27:40.570083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.65:35343 (local address 127.0.0.1:40774)
0929 20:27:40.571135 (+ 1052us) negotiation.cc:107] Waiting for socket to connect
0929 20:27:40.571194 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0929 20:27:40.571419 (+ 225us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:27:40.578227 (+ 6808us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:27:40.578232 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:27:40.578259 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:27:40.578736 (+ 477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:27:40.578749 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:27:40.580916 (+ 2167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:27:40.580923 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:27:40.583129 (+ 2206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:27:40.583143 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:27:40.585086 (+ 1943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:27:40.585093 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:27:40.585262 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:27:40.585292 (+ 30us) client_negotiation.cc:769] Sending connection context
0929 20:27:40.585370 (+ 78us) client_negotiation.cc:240] Negotiation successful
0929 20:27:40.585504 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":668,"thread_start_us":910,"threads_started":1}
I20260929 20:27:40.586793 5561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:27:40.570264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40774 (local address 127.17.53.65:35343)
0929 20:27:40.577579 (+ 7315us) server_negotiation.cc:206] Beginning negotiation
0929 20:27:40.577588 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 20:27:40.577619 (+ 31us) server_negotiation.cc:407] Connection header received
0929 20:27:40.577740 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:27:40.577750 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:27:40.578017 (+ 267us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:27:40.578947 (+ 930us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:27:40.579193 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:27:40.580767 (+ 1574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:27:40.583382 (+ 2615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:27:40.584828 (+ 1446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:27:40.585914 (+ 1086us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:27:40.586207 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0929 20:27:40.586288 (+ 81us) server_negotiation.cc:299] Negotiation successful
0929 20:27:40.586462 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7148,"thread_start_us":7032,"threads_started":1}
I20260929 20:27:40.590211 5222 tablet_service.cc:1467] Processing CreateTablet for tablet 919ba3fbfb3c4680b29d0e5b3ee50d69 (DEFAULT_TABLE table=test-table [id=8943a6a4dbaf460bbbdcbe1f7ba48213]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:27:40.590229 5353 tablet_service.cc:1467] Processing CreateTablet for tablet 919ba3fbfb3c4680b29d0e5b3ee50d69 (DEFAULT_TABLE table=test-table [id=8943a6a4dbaf460bbbdcbe1f7ba48213]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:27:40.591691 5222 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 919ba3fbfb3c4680b29d0e5b3ee50d69. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:27:40.591720 5353 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 919ba3fbfb3c4680b29d0e5b3ee50d69. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:27:40.593881 5564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:27:40.571805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.67:40291 (local address 127.0.0.1:50522)
0929 20:27:40.583290 (+ 11485us) negotiation.cc:107] Waiting for socket to connect
0929 20:27:40.583316 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0929 20:27:40.583470 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:27:40.585951 (+ 2481us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:27:40.585957 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:27:40.585995 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:27:40.586618 (+ 623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:27:40.586633 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:27:40.588938 (+ 2305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:27:40.588943 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:27:40.590925 (+ 1982us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:27:40.590943 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:27:40.593125 (+ 2182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:27:40.593132 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:27:40.593310 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:27:40.593353 (+ 43us) client_negotiation.cc:769] Sending connection context
0929 20:27:40.593441 (+ 88us) client_negotiation.cc:240] Negotiation successful
0929 20:27:40.593586 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11313,"thread_start_us":11779,"threads_started":1}
I20260929 20:27:40.594646 5565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:27:40.572423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50522 (local address 127.17.53.67:40291)
0929 20:27:40.585403 (+ 12980us) server_negotiation.cc:206] Beginning negotiation
0929 20:27:40.585410 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:27:40.585437 (+ 27us) server_negotiation.cc:407] Connection header received
0929 20:27:40.585513 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:27:40.585519 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:27:40.585695 (+ 176us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:27:40.586897 (+ 1202us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:27:40.587226 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:27:40.588776 (+ 1550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:27:40.591154 (+ 2378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:27:40.592574 (+ 1420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:27:40.592708 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:27:40.592988 (+ 280us) server_negotiation.cc:1035] Waiting for connection context
0929 20:27:40.594094 (+ 1106us) server_negotiation.cc:299] Negotiation successful
0929 20:27:40.594265 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12823,"thread_start_us":12615,"threads_started":1}
I20260929 20:27:40.598431 5483 tablet_service.cc:1467] Processing CreateTablet for tablet 919ba3fbfb3c4680b29d0e5b3ee50d69 (DEFAULT_TABLE table=test-table [id=8943a6a4dbaf460bbbdcbe1f7ba48213]), partition=RANGE (key) PARTITION UNBOUNDED
I20260929 20:27:40.599958 5483 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 919ba3fbfb3c4680b29d0e5b3ee50d69. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:27:40.604159 5566 tablet_bootstrap.cc:492] T 919ba3fbfb3c4680b29d0e5b3ee50d69 P 2393d6f0b3ca400889cd0973fae0793f: Bootstrap starting.
I20260929 20:27:40.606132 5567 tablet_bootstrap.cc:492] T 919ba3fbfb3c4680b29d0e5b3ee50d69 P ba2bfffde3d04a96b1fbe92992010591: Bootstrap starting.
I20260929 20:27:40.609490 5566 tablet_bootstrap.cc:654] T 919ba3fbfb3c4680b29d0e5b3ee50d69 P 2393d6f0b3ca400889cd0973fae0793f: Neither blocks nor log segments found. Creating new log.
I20260929 20:27:40.609699 5567 tablet_bootstrap.cc:654] T 919ba3fbfb3c4680b29d0e5b3ee50d69 P ba2bfffde3d04a96b1fbe92992010591: Neither blocks nor log segments found. Creating new log.
I20260929 20:27:40.610818 5566 log.cc:824] T 919ba3fbfb3c4680b29d0e5b3ee50d69 P 2393d6f0b3ca400889cd0973fae0793f: Log is configured to *not* fsync() on all Append() calls
I20260929 20:27:40.610817 5567 log.cc:824] T 919ba3fbfb3c4680b29d0e5b3ee50d69 P ba2bfffde3d04a96b1fbe92992010591: Log is configured to *not* fsync() on all Append() calls
I20260929 20:27:40.613575 5570 tablet_bootstrap.cc:492] T 919ba3fbfb3c4680b29d0e5b3ee50d69 P cc2b2cf56f104dad862ab3d71d0c2409: Bootstrap starting.
I20260929 20:27:40.614212 5566 tablet_bootstrap.cc:492] T 919ba3fbfb3c4680b29d0e5b3ee50d69 P 2393d6f0b3ca400889cd0973fae0793f: No bootstrap required, opened a new log
I20260929 20:27:40.614320 5567 tablet_bootstrap.cc:492] T 919ba3fbfb3c4680b29d0e5b3ee50d69 P ba2bfffde3d04a96b1fbe92992010591: No bootstrap required, opened a new log
I20260929 20:27:40.614548 5566 ts_tablet_manager.cc:1397] T 919ba3fbfb3c4680b29d0e5b3ee50d69 P 2393d6f0b3ca400889cd0973fae0793f: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.004s
I20260929 20:27:40.614601 5567 ts_tablet_manager.cc:1397] T 919ba3fbfb3c4680b29d0e5b3ee50d69 P ba2bfffde3d04a96b1fbe92992010591: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260929 20:27:40.617309 5570 tablet_bootstrap.cc:654] T 919ba3fbfb3c4680b29d0e5b3ee50d69 P cc2b2cf56f104dad862ab3d71d0c2409: Neither blocks nor log segments found. Creating new log.
I20260929 20:27:40.618587 5570 log.cc:824] T 919ba3fbfb3c4680b29d0e5b3ee50d69 P cc2b2cf56f104dad862ab3d71d0c2409: Log is configured to *not* fsync() on all Append() calls
I20260929 20:27:40.622288 5570 tablet_bootstrap.cc:492] T 919ba3fbfb3c4680b29d0e5b3ee50d69 P cc2b2cf56f104dad862ab3d71d0c2409: No bootstrap required, opened a new log
I20260929 20:27:40.622576 5570 ts_tablet_manager.cc:1397] T 919ba3fbfb3c4680b29d0e5b3ee50d69 P cc2b2cf56f104dad862ab3d71d0c2409: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260929 20:27:40.623389 5567 raft_consensus.cc:348] T 919ba3fbfb3c4680b29d0e5b3ee50d69 P ba2bfffde3d04a96b1fbe92992010591 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba2bfffde3d04a96b1fbe92992010591" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 35343 } } peers { permanent_uuid: "cc2b2cf56f104dad862ab3d71d0c2409" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 40291 } } peers { permanent_uuid: "2393d6f0b3ca400889cd0973fae0793f" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 33623 } }
I20260929 20:27:40.623836 5567 raft_consensus.cc:374] T 919ba3fbfb3c4680b29d0e5b3ee50d69 P ba2bfffde3d04a96b1fbe92992010591 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:27:40.623953 5567 raft_consensus.cc:729] T 919ba3fbfb3c4680b29d0e5b3ee50d69 P ba2bfffde3d04a96b1fbe92992010591 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba2bfffde3d04a96b1fbe92992010591, State: Initialized, Role: FOLLOWER
I20260929 20:27:40.623780 5566 raft_consensus.cc:348] T 919ba3fbfb3c4680b29d0e5b3ee50d69 P 2393d6f0b3ca400889cd0973fae0793f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba2bfffde3d04a96b1fbe92992010591" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 35343 } } peers { permanent_uuid: "cc2b2cf56f104dad862ab3d71d0c2409" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 40291 } } peers { permanent_uuid: "2393d6f0b3ca400889cd0973fae0793f" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 33623 } }
I20260929 20:27:40.624255 5566 raft_consensus.cc:374] T 919ba3fbfb3c4680b29d0e5b3ee50d69 P 2393d6f0b3ca400889cd0973fae0793f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:27:40.624408 5566 raft_consensus.cc:729] T 919ba3fbfb3c4680b29d0e5b3ee50d69 P 2393d6f0b3ca400889cd0973fae0793f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2393d6f0b3ca400889cd0973fae0793f, State: Initialized, Role: FOLLOWER
I20260929 20:27:40.624644 5567 consensus_queue.cc:260] T 919ba3fbfb3c4680b29d0e5b3ee50d69 P ba2bfffde3d04a96b1fbe92992010591 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba2bfffde3d04a96b1fbe92992010591" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 35343 } } peers { permanent_uuid: "cc2b2cf56f104dad862ab3d71d0c2409" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 40291 } } peers { permanent_uuid: "2393d6f0b3ca400889cd0973fae0793f" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 33623 } }
I20260929 20:27:40.625211 5566 consensus_queue.cc:260] T 919ba3fbfb3c4680b29d0e5b3ee50d69 P 2393d6f0b3ca400889cd0973fae0793f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba2bfffde3d04a96b1fbe92992010591" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 35343 } } peers { permanent_uuid: "cc2b2cf56f104dad862ab3d71d0c2409" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 40291 } } peers { permanent_uuid: "2393d6f0b3ca400889cd0973fae0793f" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 33623 } }
I20260929 20:27:40.626453 5567 ts_tablet_manager.cc:1428] T 919ba3fbfb3c4680b29d0e5b3ee50d69 P ba2bfffde3d04a96b1fbe92992010591: Time spent starting tablet: real 0.012s user 0.010s sys 0.000s
I20260929 20:27:40.632004 5418 heartbeater.cc:503] Master 127.17.53.126:34863 was elected leader, sending a full tablet report...
I20260929 20:27:40.633239 5566 ts_tablet_manager.cc:1428] T 919ba3fbfb3c4680b29d0e5b3ee50d69 P 2393d6f0b3ca400889cd0973fae0793f: Time spent starting tablet: real 0.019s user 0.007s sys 0.011s
I20260929 20:27:40.633019 5570 raft_consensus.cc:348] T 919ba3fbfb3c4680b29d0e5b3ee50d69 P cc2b2cf56f104dad862ab3d71d0c2409 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba2bfffde3d04a96b1fbe92992010591" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 35343 } } peers { permanent_uuid: "cc2b2cf56f104dad862ab3d71d0c2409" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 40291 } } peers { permanent_uuid: "2393d6f0b3ca400889cd0973fae0793f" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 33623 } }
I20260929 20:27:40.633481 5570 raft_consensus.cc:374] T 919ba3fbfb3c4680b29d0e5b3ee50d69 P cc2b2cf56f104dad862ab3d71d0c2409 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:27:40.633606 5570 raft_consensus.cc:729] T 919ba3fbfb3c4680b29d0e5b3ee50d69 P cc2b2cf56f104dad862ab3d71d0c2409 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc2b2cf56f104dad862ab3d71d0c2409, State: Initialized, Role: FOLLOWER
I20260929 20:27:40.634224 5570 consensus_queue.cc:260] T 919ba3fbfb3c4680b29d0e5b3ee50d69 P cc2b2cf56f104dad862ab3d71d0c2409 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba2bfffde3d04a96b1fbe92992010591" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 35343 } } peers { permanent_uuid: "cc2b2cf56f104dad862ab3d71d0c2409" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 40291 } } peers { permanent_uuid: "2393d6f0b3ca400889cd0973fae0793f" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 33623 } }
I20260929 20:27:40.635349 5548 heartbeater.cc:503] Master 127.17.53.126:34863 was elected leader, sending a full tablet report...
I20260929 20:27:40.635862 5570 ts_tablet_manager.cc:1428] T 919ba3fbfb3c4680b29d0e5b3ee50d69 P cc2b2cf56f104dad862ab3d71d0c2409: Time spent starting tablet: real 0.013s user 0.014s sys 0.000s
I20260929 20:27:40.648386 5573 raft_consensus.cc:482] T 919ba3fbfb3c4680b29d0e5b3ee50d69 P 2393d6f0b3ca400889cd0973fae0793f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 20:27:40.648728 5573 raft_consensus.cc:504] T 919ba3fbfb3c4680b29d0e5b3ee50d69 P 2393d6f0b3ca400889cd0973fae0793f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba2bfffde3d04a96b1fbe92992010591" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 35343 } } peers { permanent_uuid: "cc2b2cf56f104dad862ab3d71d0c2409" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 40291 } } peers { permanent_uuid: "2393d6f0b3ca400889cd0973fae0793f" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 33623 } }
I20260929 20:27:40.650348 5573 leader_election.cc:288] T 919ba3fbfb3c4680b29d0e5b3ee50d69 P 2393d6f0b3ca400889cd0973fae0793f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ba2bfffde3d04a96b1fbe92992010591 (127.17.53.65:35343), cc2b2cf56f104dad862ab3d71d0c2409 (127.17.53.67:40291)
I20260929 20:27:40.657444 5561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:27:40.650755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.53.66:58037 (local address 127.17.53.65:35343)
0929 20:27:40.650957 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0929 20:27:40.650962 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:27:40.651412 (+ 450us) server_negotiation.cc:407] Connection header received
0929 20:27:40.651800 (+ 388us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:27:40.651807 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:27:40.651942 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:27:40.652057 (+ 115us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:27:40.652875 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:27:40.653628 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:27:40.655535 (+ 1907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:27:40.656558 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:27:40.656738 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:27:40.656808 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0929 20:27:40.657091 (+ 283us) server_negotiation.cc:299] Negotiation successful
0929 20:27:40.657211 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260929 20:27:40.657498 5575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:27:40.650626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.65:35343 (local address 127.17.53.66:58037)
0929 20:27:40.651312 (+ 686us) negotiation.cc:107] Waiting for socket to connect
0929 20:27:40.651332 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0929 20:27:40.651571 (+ 239us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:27:40.652224 (+ 653us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:27:40.652233 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:27:40.652262 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:27:40.652726 (+ 464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:27:40.652740 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:27:40.653763 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:27:40.653770 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:27:40.655342 (+ 1572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:27:40.655354 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:27:40.656751 (+ 1397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:27:40.656756 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:27:40.656938 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:27:40.656962 (+ 24us) client_negotiation.cc:769] Sending connection context
0929 20:27:40.657064 (+ 102us) client_negotiation.cc:240] Negotiation successful
0929 20:27:40.657186 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":531,"thread_start_us":349,"threads_started":1}
I20260929 20:27:40.658493 5242 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "919ba3fbfb3c4680b29d0e5b3ee50d69" candidate_uuid: "2393d6f0b3ca400889cd0973fae0793f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba2bfffde3d04a96b1fbe92992010591" is_pre_election: true
I20260929 20:27:40.658738 5565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:27:40.651672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.17.53.66:47013 (local address 127.17.53.67:40291)
0929 20:27:40.651866 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0929 20:27:40.651873 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:27:40.653172 (+ 1299us) server_negotiation.cc:407] Connection header received
0929 20:27:40.653357 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:27:40.653362 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:27:40.653431 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:27:40.653523 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0929 20:27:40.654112 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:27:40.654833 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:27:40.656763 (+ 1930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:27:40.657905 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:27:40.658073 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:27:40.658139 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0929 20:27:40.658389 (+ 250us) server_negotiation.cc:299] Negotiation successful
0929 20:27:40.658518 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260929 20:27:40.658756 5576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:27:40.651437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.67:40291 (local address 127.17.53.66:47013)
0929 20:27:40.653068 (+ 1631us) negotiation.cc:107] Waiting for socket to connect
0929 20:27:40.653082 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0929 20:27:40.653221 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:27:40.653549 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:27:40.653553 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:27:40.653576 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0929 20:27:40.653951 (+ 375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:27:40.653964 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:27:40.655001 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:27:40.655006 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:27:40.656553 (+ 1547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:27:40.656565 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:27:40.658073 (+ 1508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:27:40.658079 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:27:40.658240 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:27:40.658262 (+ 22us) client_negotiation.cc:769] Sending connection context
0929 20:27:40.658364 (+ 102us) client_negotiation.cc:240] Negotiation successful
0929 20:27:40.658481 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1516,"thread_start_us":804,"threads_started":1}
I20260929 20:27:40.659080 5242 raft_consensus.cc:2454] T 919ba3fbfb3c4680b29d0e5b3ee50d69 P ba2bfffde3d04a96b1fbe92992010591 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2393d6f0b3ca400889cd0973fae0793f in term 0.
I20260929 20:27:40.659780 5308 leader_election.cc:302] T 919ba3fbfb3c4680b29d0e5b3ee50d69 P 2393d6f0b3ca400889cd0973fae0793f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2393d6f0b3ca400889cd0973fae0793f, ba2bfffde3d04a96b1fbe92992010591; no voters:
I20260929 20:27:40.659878 5503 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "919ba3fbfb3c4680b29d0e5b3ee50d69" candidate_uuid: "2393d6f0b3ca400889cd0973fae0793f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc2b2cf56f104dad862ab3d71d0c2409" is_pre_election: true
I20260929 20:27:40.660243 5573 raft_consensus.cc:2781] T 919ba3fbfb3c4680b29d0e5b3ee50d69 P 2393d6f0b3ca400889cd0973fae0793f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 20:27:40.660398 5573 raft_consensus.cc:482] T 919ba3fbfb3c4680b29d0e5b3ee50d69 P 2393d6f0b3ca400889cd0973fae0793f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 20:27:40.660410 5503 raft_consensus.cc:2454] T 919ba3fbfb3c4680b29d0e5b3ee50d69 P cc2b2cf56f104dad862ab3d71d0c2409 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2393d6f0b3ca400889cd0973fae0793f in term 0.
I20260929 20:27:40.660523 5573 raft_consensus.cc:3037] T 919ba3fbfb3c4680b29d0e5b3ee50d69 P 2393d6f0b3ca400889cd0973fae0793f [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:27:40.662766 5573 raft_consensus.cc:504] T 919ba3fbfb3c4680b29d0e5b3ee50d69 P 2393d6f0b3ca400889cd0973fae0793f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba2bfffde3d04a96b1fbe92992010591" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 35343 } } peers { permanent_uuid: "cc2b2cf56f104dad862ab3d71d0c2409" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 40291 } } peers { permanent_uuid: "2393d6f0b3ca400889cd0973fae0793f" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 33623 } }
I20260929 20:27:40.663388 5573 leader_election.cc:288] T 919ba3fbfb3c4680b29d0e5b3ee50d69 P 2393d6f0b3ca400889cd0973fae0793f [CANDIDATE]: Term 1 election: Requested vote from peers ba2bfffde3d04a96b1fbe92992010591 (127.17.53.65:35343), cc2b2cf56f104dad862ab3d71d0c2409 (127.17.53.67:40291)
I20260929 20:27:40.663829 5242 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "919ba3fbfb3c4680b29d0e5b3ee50d69" candidate_uuid: "2393d6f0b3ca400889cd0973fae0793f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba2bfffde3d04a96b1fbe92992010591"
I20260929 20:27:40.663875 5503 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "919ba3fbfb3c4680b29d0e5b3ee50d69" candidate_uuid: "2393d6f0b3ca400889cd0973fae0793f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cc2b2cf56f104dad862ab3d71d0c2409"
I20260929 20:27:40.664042 5242 raft_consensus.cc:3037] T 919ba3fbfb3c4680b29d0e5b3ee50d69 P ba2bfffde3d04a96b1fbe92992010591 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:27:40.664047 5503 raft_consensus.cc:3037] T 919ba3fbfb3c4680b29d0e5b3ee50d69 P cc2b2cf56f104dad862ab3d71d0c2409 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:27:40.666182 5503 raft_consensus.cc:2454] T 919ba3fbfb3c4680b29d0e5b3ee50d69 P cc2b2cf56f104dad862ab3d71d0c2409 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2393d6f0b3ca400889cd0973fae0793f in term 1.
I20260929 20:27:40.666182 5242 raft_consensus.cc:2454] T 919ba3fbfb3c4680b29d0e5b3ee50d69 P ba2bfffde3d04a96b1fbe92992010591 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2393d6f0b3ca400889cd0973fae0793f in term 1.
I20260929 20:27:40.666535 5308 leader_election.cc:302] T 919ba3fbfb3c4680b29d0e5b3ee50d69 P 2393d6f0b3ca400889cd0973fae0793f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2393d6f0b3ca400889cd0973fae0793f, cc2b2cf56f104dad862ab3d71d0c2409; no voters:
I20260929 20:27:40.666842 5573 raft_consensus.cc:2781] T 919ba3fbfb3c4680b29d0e5b3ee50d69 P 2393d6f0b3ca400889cd0973fae0793f [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:27:40.667356 5573 raft_consensus.cc:686] T 919ba3fbfb3c4680b29d0e5b3ee50d69 P 2393d6f0b3ca400889cd0973fae0793f [term 1 LEADER]: Becoming Leader. State: Replica: 2393d6f0b3ca400889cd0973fae0793f, State: Running, Role: LEADER
I20260929 20:27:40.667871 5573 consensus_queue.cc:237] T 919ba3fbfb3c4680b29d0e5b3ee50d69 P 2393d6f0b3ca400889cd0973fae0793f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba2bfffde3d04a96b1fbe92992010591" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 35343 } } peers { permanent_uuid: "cc2b2cf56f104dad862ab3d71d0c2409" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 40291 } } peers { permanent_uuid: "2393d6f0b3ca400889cd0973fae0793f" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 33623 } }
I20260929 20:27:40.672441 5100 catalog_manager.cc:5495] T 919ba3fbfb3c4680b29d0e5b3ee50d69 P 2393d6f0b3ca400889cd0973fae0793f reported cstate change: term changed from 0 to 1, leader changed from <none> to 2393d6f0b3ca400889cd0973fae0793f (127.17.53.66). New cstate: current_term: 1 leader_uuid: "2393d6f0b3ca400889cd0973fae0793f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ba2bfffde3d04a96b1fbe92992010591" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 35343 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cc2b2cf56f104dad862ab3d71d0c2409" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 40291 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2393d6f0b3ca400889cd0973fae0793f" member_type: VOTER last_known_addr { host: "127.17.53.66" port: 33623 } health_report { overall_health: HEALTHY } } }
I20260929 20:27:40.704177 5563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:27:40.699008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35966 (local address 127.17.53.66:33623)
0929 20:27:40.699227 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0929 20:27:40.699233 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:27:40.699253 (+ 20us) server_negotiation.cc:407] Connection header received
0929 20:27:40.699364 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:27:40.699368 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:27:40.699453 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:27:40.699606 (+ 153us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 20:27:40.699995 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:27:40.700689 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:27:40.702050 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:27:40.702475 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:27:40.702611 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:27:40.703133 (+ 522us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 20:27:40.703622 (+ 489us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 20:27:40.703789 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0929 20:27:40.703851 (+ 62us) server_negotiation.cc:299] Negotiation successful
0929 20:27:40.703925 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260929 20:27:40.712354 5242 raft_consensus.cc:1261] T 919ba3fbfb3c4680b29d0e5b3ee50d69 P ba2bfffde3d04a96b1fbe92992010591 [term 1 FOLLOWER]: Refusing update from remote peer 2393d6f0b3ca400889cd0973fae0793f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:27:40.712301 5503 raft_consensus.cc:1261] T 919ba3fbfb3c4680b29d0e5b3ee50d69 P cc2b2cf56f104dad862ab3d71d0c2409 [term 1 FOLLOWER]: Refusing update from remote peer 2393d6f0b3ca400889cd0973fae0793f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:27:40.713356 5577 consensus_queue.cc:1035] T 919ba3fbfb3c4680b29d0e5b3ee50d69 P 2393d6f0b3ca400889cd0973fae0793f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba2bfffde3d04a96b1fbe92992010591" member_type: VOTER last_known_addr { host: "127.17.53.65" port: 35343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:27:40.713709 5573 consensus_queue.cc:1035] T 919ba3fbfb3c4680b29d0e5b3ee50d69 P 2393d6f0b3ca400889cd0973fae0793f [LEADER]: Connected to new peer: Peer: permanent_uuid: "cc2b2cf56f104dad862ab3d71d0c2409" member_type: VOTER last_known_addr { host: "127.17.53.67" port: 40291 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:27:40.726902 5581 mvcc.cc:205] Tried to move back new op lower bound from 7334763154267701248 to 7334763154099961856. Current Snapshot: MvccSnapshot[applied={T|T < 7334763154267701248}]
I20260929 20:27:40.727005 5582 mvcc.cc:205] Tried to move back new op lower bound from 7334763154267701248 to 7334763154099961856. Current Snapshot: MvccSnapshot[applied={T|T < 7334763154267701248}]
I20260929 20:27:40.739645 5101 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:59580:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 20:27:40.739919 5101 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:59580:
table { table_name: "test-table" } modify_external_catalogs: true
I20260929 20:27:40.743650 5101 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3f1b9c4d140c4dcd805e1b439f5490f5: Sending DeleteTablet for 3 replicas of tablet 919ba3fbfb3c4680b29d0e5b3ee50d69
I20260929 20:27:40.744874 5353 tablet_service.cc:1514] Processing DeleteTablet for tablet 919ba3fbfb3c4680b29d0e5b3ee50d69 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:27:40 UTC) from {username='slave'} at 127.0.0.1:35964
I20260929 20:27:40.744938 5483 tablet_service.cc:1514] Processing DeleteTablet for tablet 919ba3fbfb3c4680b29d0e5b3ee50d69 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:27:40 UTC) from {username='slave'} at 127.0.0.1:50522
I20260929 20:27:40.745950 17621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu pid 5160
I20260929 20:27:40.745899 5222 tablet_service.cc:1514] Processing DeleteTablet for tablet 919ba3fbfb3c4680b29d0e5b3ee50d69 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 20:27:40 UTC) from {username='slave'} at 127.0.0.1:40774
I20260929 20:27:40.756596 5590 tablet_replica.cc:330] T 919ba3fbfb3c4680b29d0e5b3ee50d69 P 2393d6f0b3ca400889cd0973fae0793f: stopping tablet replica
I20260929 20:27:40.757376 5590 raft_consensus.cc:2229] T 919ba3fbfb3c4680b29d0e5b3ee50d69 P 2393d6f0b3ca400889cd0973fae0793f [term 1 LEADER]: Raft consensus shutting down.
I20260929 20:27:40.758245 5590 raft_consensus.cc:2258] T 919ba3fbfb3c4680b29d0e5b3ee50d69 P 2393d6f0b3ca400889cd0973fae0793f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:27:40.758965 5591 tablet_replica.cc:330] T 919ba3fbfb3c4680b29d0e5b3ee50d69 P cc2b2cf56f104dad862ab3d71d0c2409: stopping tablet replica
I20260929 20:27:40.759498 5591 raft_consensus.cc:2229] T 919ba3fbfb3c4680b29d0e5b3ee50d69 P cc2b2cf56f104dad862ab3d71d0c2409 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 20:27:40.762032 5590 ts_tablet_manager.cc:1905] T 919ba3fbfb3c4680b29d0e5b3ee50d69 P 2393d6f0b3ca400889cd0973fae0793f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:27:40.762634 5591 raft_consensus.cc:2258] T 919ba3fbfb3c4680b29d0e5b3ee50d69 P cc2b2cf56f104dad862ab3d71d0c2409 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:27:40.765002 5592 tablet_replica.cc:330] T 919ba3fbfb3c4680b29d0e5b3ee50d69 P ba2bfffde3d04a96b1fbe92992010591: stopping tablet replica
I20260929 20:27:40.765514 5592 raft_consensus.cc:2229] T 919ba3fbfb3c4680b29d0e5b3ee50d69 P ba2bfffde3d04a96b1fbe92992010591 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 20:27:40.766067 5592 raft_consensus.cc:2258] T 919ba3fbfb3c4680b29d0e5b3ee50d69 P ba2bfffde3d04a96b1fbe92992010591 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 20:27:40.766148 5591 ts_tablet_manager.cc:1905] T 919ba3fbfb3c4680b29d0e5b3ee50d69 P cc2b2cf56f104dad862ab3d71d0c2409: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:27:40.770327 5561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:27:40.747165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40778 (local address 127.17.53.65:35343)
0929 20:27:40.747345 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0929 20:27:40.747352 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:27:40.757782 (+ 10430us) server_negotiation.cc:407] Connection header received
0929 20:27:40.758312 (+ 530us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:27:40.758319 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:27:40.758460 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:27:40.758572 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:27:40.759313 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:27:40.760541 (+ 1228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:27:40.762411 (+ 1870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:27:40.763129 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:27:40.763242 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:27:40.768546 (+ 5304us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:27:40.768557 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:27:40.768574 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:27:40.769372 (+ 798us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:27:40.769434 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:27:40.769460 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:27:40.769560 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0929 20:27:40.769825 (+ 265us) server_negotiation.cc:299] Negotiation successful
0929 20:27:40.769976 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 20:27:40.771207 5590 ts_tablet_manager.cc:1918] T 919ba3fbfb3c4680b29d0e5b3ee50d69 P 2393d6f0b3ca400889cd0973fae0793f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 20:27:40.771466 5590 log.cc:1196] T 919ba3fbfb3c4680b29d0e5b3ee50d69 P 2393d6f0b3ca400889cd0973fae0793f: Deleting WAL directory at /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790713339161492-17621-0/minicluster-data/ts-1/wal/wals/919ba3fbfb3c4680b29d0e5b3ee50d69
I20260929 20:27:40.771801 5282 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:27:40.772197 5590 ts_tablet_manager.cc:1939] T 919ba3fbfb3c4680b29d0e5b3ee50d69 P 2393d6f0b3ca400889cd0973fae0793f: Deleting consensus metadata
I20260929 20:27:40.773286 5591 ts_tablet_manager.cc:1918] T 919ba3fbfb3c4680b29d0e5b3ee50d69 P cc2b2cf56f104dad862ab3d71d0c2409: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260929 20:27:40.773471 5591 log.cc:1196] T 919ba3fbfb3c4680b29d0e5b3ee50d69 P cc2b2cf56f104dad862ab3d71d0c2409: Deleting WAL directory at /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790713339161492-17621-0/minicluster-data/ts-2/wal/wals/919ba3fbfb3c4680b29d0e5b3ee50d69
I20260929 20:27:40.774026 5591 ts_tablet_manager.cc:1939] T 919ba3fbfb3c4680b29d0e5b3ee50d69 P cc2b2cf56f104dad862ab3d71d0c2409: Deleting consensus metadata
I20260929 20:27:40.774593 5085 catalog_manager.cc:4841] TS 2393d6f0b3ca400889cd0973fae0793f (127.17.53.66:33623): tablet 919ba3fbfb3c4680b29d0e5b3ee50d69 (table test-table [id=8943a6a4dbaf460bbbdcbe1f7ba48213]) successfully deleted
I20260929 20:27:40.776007 5087 catalog_manager.cc:4841] TS cc2b2cf56f104dad862ab3d71d0c2409 (127.17.53.67:40291): tablet 919ba3fbfb3c4680b29d0e5b3ee50d69 (table test-table [id=8943a6a4dbaf460bbbdcbe1f7ba48213]) successfully deleted
I20260929 20:27:40.776548 5592 ts_tablet_manager.cc:1905] T 919ba3fbfb3c4680b29d0e5b3ee50d69 P ba2bfffde3d04a96b1fbe92992010591: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 20:27:40.867053 17621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu with pid 5160
I20260929 20:27:40.884902 17621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu pid 5291
W20260929 20:27:40.884933 5087 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.17.53.65:35343 (error 108)
I20260929 20:27:40.888242 5564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:27:40.887712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.65:35343 (local address 127.0.0.1:40786)
0929 20:27:40.887932 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0929 20:27:40.888055 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.53.65:35343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260929 20:27:40.888551 5087 catalog_manager.cc:4584] TS ba2bfffde3d04a96b1fbe92992010591 (127.17.53.65:35343): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 919ba3fbfb3c4680b29d0e5b3ee50d69: Network error: Client connection negotiation failed: client connection to 127.17.53.65:35343: connect: Connection refused (error 111)
I20260929 20:27:40.893060 5563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:27:40.886046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35974 (local address 127.17.53.66:33623)
0929 20:27:40.886228 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0929 20:27:40.886235 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:27:40.886272 (+ 37us) server_negotiation.cc:407] Connection header received
0929 20:27:40.886342 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:27:40.886348 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:27:40.886451 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:27:40.886572 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:27:40.887187 (+ 615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:27:40.888255 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:27:40.889807 (+ 1552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:27:40.890239 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:27:40.890378 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:27:40.891530 (+ 1152us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:27:40.891539 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:27:40.891553 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:27:40.892206 (+ 653us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:27:40.892262 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:27:40.892282 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:27:40.892430 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0929 20:27:40.892606 (+ 176us) server_negotiation.cc:299] Negotiation successful
0929 20:27:40.892708 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260929 20:27:40.893734 5413 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:27:40.942415 5564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:27:40.941916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.65:35343 (local address 127.0.0.1:40790)
0929 20:27:40.942141 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0929 20:27:40.942250 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.53.65:35343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260929 20:27:40.943480 5564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:27:40.943121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.65:35343 (local address 127.0.0.1:40800)
0929 20:27:40.943280 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0929 20:27:40.943360 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.53.65:35343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260929 20:27:41.003924 5564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:27:41.003409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.65:35343 (local address 127.0.0.1:40814)
0929 20:27:41.003649 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0929 20:27:41.003741 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.53.65:35343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100}
I20260929 20:27:41.004964 5564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:27:41.004627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.65:35343 (local address 127.0.0.1:40830)
0929 20:27:41.004779 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0929 20:27:41.004854 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.53.65:35343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260929 20:27:41.007557 17621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu with pid 5291
I20260929 20:27:41.026783 17621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu pid 5421
I20260929 20:27:41.034065 5565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:27:41.027614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50534 (local address 127.17.53.67:40291)
0929 20:27:41.027849 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0929 20:27:41.027855 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:27:41.028107 (+ 252us) server_negotiation.cc:407] Connection header received
0929 20:27:41.028357 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:27:41.028369 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:27:41.028464 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:27:41.028577 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:27:41.029027 (+ 450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:27:41.029778 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:27:41.031148 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:27:41.031545 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:27:41.031683 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:27:41.032745 (+ 1062us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:27:41.032756 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:27:41.032775 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:27:41.033359 (+ 584us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:27:41.033405 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:27:41.033424 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:27:41.033567 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0929 20:27:41.033655 (+ 88us) server_negotiation.cc:299] Negotiation successful
0929 20:27:41.033761 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260929 20:27:41.034714 5543 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:27:41.085714 5564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:27:41.085213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.65:35343 (local address 127.0.0.1:40838)
0929 20:27:41.085452 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0929 20:27:41.085540 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.53.65:35343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103}
I20260929 20:27:41.086782 5564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:27:41.086365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.65:35343 (local address 127.0.0.1:40840)
0929 20:27:41.086603 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0929 20:27:41.086676 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.53.65:35343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260929 20:27:41.130712 17621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu with pid 5421
I20260929 20:27:41.148010 17621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu pid 5070
I20260929 20:27:41.149143 5131 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:27:41.259660 17621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu with pid 5070
I20260929 20:27:41.259730 5564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:27:41.258978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.65:35343 (local address 127.0.0.1:40850)
0929 20:27:41.259441 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0929 20:27:41.259543 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.53.65:35343: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":311}
2026-09-29T20:27:41Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (2808 ms)
[----------] 6 tests from EncryptionPolicyTest (21965 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790713339161492-17621-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 20:27:41 dist-test-slave-zbnc krb5kdc[5601](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 20:27:41 dist-test-slave-zbnc krb5kdc[5601](info): set up 2 sockets
Sep 29 20:27:41 dist-test-slave-zbnc krb5kdc[5601](info): commencing operation
krb5kdc: starting...
W20260929 20:27:42.376734 17621 mini_kdc.cc:121] Time spent starting KDC: real 1.074s user 0.006s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790713339161492-17621-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790713339161492-17621-0/krb5kdc/test-user.keytab.
Sep 29 20:27:42 dist-test-slave-zbnc krb5kdc[5601](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713662, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T20:27:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T20:27:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790713339161492-17621-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790713339161492-17621-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 20:27:42.639094 17621 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
/tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790713339161492-17621-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790713339161492-17621-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790713339161492-17621-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790713339161492-17621-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36163
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790713339161492-17621-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43301
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36163
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790713339161492-17621-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790713339161492-17621-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790713339161492-17621-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:27:42.986632 5614 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:27:42.986981 5614 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:27:42.987162 5614 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:27:42.994302 5614 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:27:42.994408 5614 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:27:42.994514 5614 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:27:42.994614 5614 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:27:42.994678 5614 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:27:43.004771 5614 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43301
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790713339161492-17621-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790713339161492-17621-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36163
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790713339161492-17621-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36163
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790713339161492-17621-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790713339161492-17621-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:27:43.006028 5614 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:27:43.007393 5614 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:27:43.019022 5620 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:27:43.019002 5619 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:27:43.019474 5622 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:27:43.019901 5614 server_base.cc:1034] running on GCE node
I20260929 20:27:43.020679 5614 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:27:43.022089 5614 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:27:43.023288 5614 hybrid_clock.cc:648] HybridClock initialized: now 1790713663023252 us; error 28 us; skew 500 ppm
I20260929 20:27:43.023663 5614 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:27:43 dist-test-slave-zbnc krb5kdc[5601](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713663, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:27:43.027048 5614 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 20:27:43.028524 5614 webserver.cc:460] Webserver started at http://127.0.0.1:33543/ using document root <none> and password file <none>
I20260929 20:27:43.029218 5614 fs_manager.cc:359] Metadata directory not provided
I20260929 20:27:43.029357 5614 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:27:43.029683 5614 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:27:43.032898 5614 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790713339161492-17621-0/minicluster-data/master-0/data/instance:
uuid: "391d0a58a2f74b44935f811089fd4f99"
format_stamp: "Formatted at 2026-09-29 20:27:43 on dist-test-slave-zbnc"
server_key: "1f562351b67712b44d935e178f42c936"
server_key_iv: "0028d1d5bfe705d5bbc46dfbc6db3b9d"
server_key_version: "encryptionkey@0"
I20260929 20:27:43.033763 5614 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790713339161492-17621-0/minicluster-data/master-0/wal/instance:
uuid: "391d0a58a2f74b44935f811089fd4f99"
format_stamp: "Formatted at 2026-09-29 20:27:43 on dist-test-slave-zbnc"
server_key: "1f562351b67712b44d935e178f42c936"
server_key_iv: "0028d1d5bfe705d5bbc46dfbc6db3b9d"
server_key_version: "encryptionkey@0"
I20260929 20:27:43.039191 5614 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260929 20:27:43.042685 5629 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:27:43.044103 5614 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 20:27:43.044291 5614 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790713339161492-17621-0/minicluster-data/master-0/data,/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790713339161492-17621-0/minicluster-data/master-0/wal
uuid: "391d0a58a2f74b44935f811089fd4f99"
format_stamp: "Formatted at 2026-09-29 20:27:43 on dist-test-slave-zbnc"
server_key: "1f562351b67712b44d935e178f42c936"
server_key_iv: "0028d1d5bfe705d5bbc46dfbc6db3b9d"
server_key_version: "encryptionkey@0"
I20260929 20:27:43.044492 5614 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790713339161492-17621-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790713339161492-17621-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790713339161492-17621-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:27:43.064946 5614 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:27:43.069383 5614 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:27:43.069679 5614 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:27:43.089053 5614 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36163
I20260929 20:27:43.089084 5680 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36163 every 8 connection(s)
I20260929 20:27:43.090445 5614 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790713339161492-17621-0/minicluster-data/master-0/data/info.pb
I20260929 20:27:43.094220 5681 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:27:43.094349 17621 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu as pid 5614
I20260929 20:27:43.094578 17621 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790713339161492-17621-0/minicluster-data/master-0/wal/instance
I20260929 20:27:43.095317 17621 external_mini_cluster.cc:1428] Setting key 357c097b9c5d389e67b9743da568e31c
I20260929 20:27:43.107023 5681 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 391d0a58a2f74b44935f811089fd4f99: Bootstrap starting.
Sep 29 20:27:43 dist-test-slave-zbnc krb5kdc[5601](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713662, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260929 20:27:43.110013 5681 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 391d0a58a2f74b44935f811089fd4f99: Neither blocks nor log segments found. Creating new log.
I20260929 20:27:43.111131 5681 log.cc:824] T 00000000000000000000000000000000 P 391d0a58a2f74b44935f811089fd4f99: Log is configured to *not* fsync() on all Append() calls
I20260929 20:27:43.114416 5681 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 391d0a58a2f74b44935f811089fd4f99: No bootstrap required, opened a new log
I20260929 20:27:43.119280 5684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:27:43.097452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41977 (local address 127.0.0.1:36163)
0929 20:27:43.098236 (+ 784us) server_negotiation.cc:206] Beginning negotiation
0929 20:27:43.098252 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0929 20:27:43.098303 (+ 51us) server_negotiation.cc:407] Connection header received
0929 20:27:43.100710 (+ 2407us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:27:43.100737 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:27:43.101142 (+ 405us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:27:43.101566 (+ 424us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:27:43.103162 (+ 1596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:27:43.104485 (+ 1323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:27:43.105794 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:27:43.106458 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:27:43.106674 (+ 216us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:27:43.110831 (+ 4157us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:27:43.110839 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:27:43.110898 (+ 59us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:27:43.110951 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:27:43.114692 (+ 3741us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:27:43.115513 (+ 821us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:27:43.115521 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:27:43.115529 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:27:43.115741 (+ 212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:27:43.116206 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:27:43.116212 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:27:43.116215 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:27:43.116933 (+ 718us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:27:43.117270 (+ 337us) server_negotiation.cc:1035] Waiting for connection context
0929 20:27:43.117582 (+ 312us) server_negotiation.cc:299] Negotiation successful
0929 20:27:43.117918 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":537,"thread_start_us":326,"threads_started":1}
I20260929 20:27:43.123425 17621 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 20:27:43.124281 5681 raft_consensus.cc:348] T 00000000000000000000000000000000 P 391d0a58a2f74b44935f811089fd4f99 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "391d0a58a2f74b44935f811089fd4f99" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36163 } }
I20260929 20:27:43.124667 5681 raft_consensus.cc:374] T 00000000000000000000000000000000 P 391d0a58a2f74b44935f811089fd4f99 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:27:43.124800 5681 raft_consensus.cc:729] T 00000000000000000000000000000000 P 391d0a58a2f74b44935f811089fd4f99 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 391d0a58a2f74b44935f811089fd4f99, State: Initialized, Role: FOLLOWER
I20260929 20:27:43.125406 5681 consensus_queue.cc:260] T 00000000000000000000000000000000 P 391d0a58a2f74b44935f811089fd4f99 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "391d0a58a2f74b44935f811089fd4f99" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36163 } }
I20260929 20:27:43.125667 5681 raft_consensus.cc:388] T 00000000000000000000000000000000 P 391d0a58a2f74b44935f811089fd4f99 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 20:27:43.125833 5681 raft_consensus.cc:482] T 00000000000000000000000000000000 P 391d0a58a2f74b44935f811089fd4f99 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 20:27:43.126012 5681 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 391d0a58a2f74b44935f811089fd4f99 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:27:43.128648 5681 raft_consensus.cc:504] T 00000000000000000000000000000000 P 391d0a58a2f74b44935f811089fd4f99 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "391d0a58a2f74b44935f811089fd4f99" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36163 } }
I20260929 20:27:43.129323 5681 leader_election.cc:302] T 00000000000000000000000000000000 P 391d0a58a2f74b44935f811089fd4f99 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 391d0a58a2f74b44935f811089fd4f99; no voters:
I20260929 20:27:43.130115 5681 leader_election.cc:288] T 00000000000000000000000000000000 P 391d0a58a2f74b44935f811089fd4f99 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 20:27:43.130244 5691 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 391d0a58a2f74b44935f811089fd4f99 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:27:43.131985 5691 raft_consensus.cc:686] T 00000000000000000000000000000000 P 391d0a58a2f74b44935f811089fd4f99 [term 1 LEADER]: Becoming Leader. State: Replica: 391d0a58a2f74b44935f811089fd4f99, State: Running, Role: LEADER
I20260929 20:27:43.132710 5691 consensus_queue.cc:237] T 00000000000000000000000000000000 P 391d0a58a2f74b44935f811089fd4f99 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "391d0a58a2f74b44935f811089fd4f99" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36163 } }
I20260929 20:27:43.132901 5681 sys_catalog.cc:564] T 00000000000000000000000000000000 P 391d0a58a2f74b44935f811089fd4f99 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:27:43.136305 5693 sys_catalog.cc:455] T 00000000000000000000000000000000 P 391d0a58a2f74b44935f811089fd4f99 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 391d0a58a2f74b44935f811089fd4f99. Latest consensus state: current_term: 1 leader_uuid: "391d0a58a2f74b44935f811089fd4f99" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "391d0a58a2f74b44935f811089fd4f99" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36163 } } }
I20260929 20:27:43.136636 5693 sys_catalog.cc:458] T 00000000000000000000000000000000 P 391d0a58a2f74b44935f811089fd4f99 [sys.catalog]: This master's current role is: LEADER
I20260929 20:27:43.136848 5692 sys_catalog.cc:455] T 00000000000000000000000000000000 P 391d0a58a2f74b44935f811089fd4f99 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "391d0a58a2f74b44935f811089fd4f99" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "391d0a58a2f74b44935f811089fd4f99" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36163 } } }
I20260929 20:27:43.137187 5692 sys_catalog.cc:458] T 00000000000000000000000000000000 P 391d0a58a2f74b44935f811089fd4f99 [sys.catalog]: This master's current role is: LEADER
I20260929 20:27:43.151362 5684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:27:43.128566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33853 (local address 127.0.0.1:36163)
0929 20:27:43.128791 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0929 20:27:43.128800 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 20:27:43.129389 (+ 589us) server_negotiation.cc:407] Connection header received
0929 20:27:43.129638 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:27:43.129642 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:27:43.129703 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:27:43.129821 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:27:43.131718 (+ 1897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:27:43.132730 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:27:43.134625 (+ 1895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:27:43.135844 (+ 1219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:27:43.136064 (+ 220us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:27:43.140583 (+ 4519us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:27:43.140594 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:27:43.140602 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:27:43.140656 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:27:43.145387 (+ 4731us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:27:43.146692 (+ 1305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:27:43.146701 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:27:43.146707 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:27:43.146912 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:27:43.147647 (+ 735us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:27:43.147655 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:27:43.147661 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:27:43.148202 (+ 541us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:27:43.148517 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0929 20:27:43.150356 (+ 1839us) server_negotiation.cc:299] Negotiation successful
0929 20:27:43.150564 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260929 20:27:43.151464 5699 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:27:43.156554 5699 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:27:43.165973 5699 catalog_manager.cc:1347] Generated new cluster ID: c456508526614cefb4e2949e6a4bad8b
I20260929 20:27:43.166085 5699 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:27:43.187481 5699 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 20:27:43.189018 5699 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:27:43.201329 5699 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 391d0a58a2f74b44935f811089fd4f99: Generated new TSK 0
I20260929 20:27:43.202244 5699 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 20:27:43.213534 17621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu pid 5614
I20260929 20:27:43.215617 5676 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:27:43.307245 17621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu with pid 5614
W20260929 20:27:43.325047 5599 connection.cc:535] client connection to 127.0.0.1:36163 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36163): Connection reset by peer (error 104)
2026-09-29T20:27:43Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (2041 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790713339161492-17621-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 20:27:43 dist-test-slave-zbnc krb5kdc[5713](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 20:27:43 dist-test-slave-zbnc krb5kdc[5713](info): set up 2 sockets
Sep 29 20:27:43 dist-test-slave-zbnc krb5kdc[5713](info): commencing operation
krb5kdc: starting...
W20260929 20:27:44.440675 17621 mini_kdc.cc:121] Time spent starting KDC: real 1.097s user 0.000s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790713339161492-17621-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790713339161492-17621-0/krb5kdc/test-user.keytab.
Sep 29 20:27:44 dist-test-slave-zbnc krb5kdc[5713](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713664, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T20:27:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T20:27:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790713339161492-17621-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790713339161492-17621-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 20:27:44.702729 17621 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
/tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790713339161492-17621-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790713339161492-17621-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790713339161492-17621-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790713339161492-17621-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33539
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790713339161492-17621-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46773
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33539
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790713339161492-17621-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790713339161492-17621-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790713339161492-17621-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:27:45.030220 5726 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:27:45.030511 5726 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:27:45.030632 5726 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:27:45.037990 5726 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:27:45.038130 5726 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:27:45.038231 5726 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:27:45.038339 5726 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:27:45.038414 5726 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:27:45.049074 5726 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46773
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790713339161492-17621-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790713339161492-17621-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33539
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790713339161492-17621-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33539
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790713339161492-17621-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790713339161492-17621-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:27:45.050623 5726 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:27:45.052627 5726 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:27:45.065841 5731 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:27:45.065860 5734 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:27:45.065883 5732 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:27:45.066725 5726 server_base.cc:1034] running on GCE node
I20260929 20:27:45.067509 5726 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:27:45.068892 5726 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:27:45.070130 5726 hybrid_clock.cc:648] HybridClock initialized: now 1790713665070088 us; error 38 us; skew 500 ppm
I20260929 20:27:45.070492 5726 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:27:45 dist-test-slave-zbnc krb5kdc[5713](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713665, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:27:45.073629 5726 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 20:27:45.075150 5726 webserver.cc:460] Webserver started at http://127.0.0.1:37813/ using document root <none> and password file <none>
I20260929 20:27:45.075820 5726 fs_manager.cc:359] Metadata directory not provided
I20260929 20:27:45.075964 5726 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:27:45.076269 5726 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:27:45.079437 5726 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790713339161492-17621-0/minicluster-data/master-0/data/instance:
uuid: "5ffad14b1f2d40f8a08bde8d66b1a0af"
format_stamp: "Formatted at 2026-09-29 20:27:45 on dist-test-slave-zbnc"
server_key: "b41ed46d4a46c8b4ae292ac1ce9aac39"
server_key_iv: "e9782158bd70c990298cfaf5d7e6837a"
server_key_version: "encryptionkey@0"
I20260929 20:27:45.080214 5726 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790713339161492-17621-0/minicluster-data/master-0/wal/instance:
uuid: "5ffad14b1f2d40f8a08bde8d66b1a0af"
format_stamp: "Formatted at 2026-09-29 20:27:45 on dist-test-slave-zbnc"
server_key: "b41ed46d4a46c8b4ae292ac1ce9aac39"
server_key_iv: "e9782158bd70c990298cfaf5d7e6837a"
server_key_version: "encryptionkey@0"
I20260929 20:27:45.085337 5726 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 20:27:45.088830 5741 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:27:45.090132 5726 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260929 20:27:45.090332 5726 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790713339161492-17621-0/minicluster-data/master-0/data,/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790713339161492-17621-0/minicluster-data/master-0/wal
uuid: "5ffad14b1f2d40f8a08bde8d66b1a0af"
format_stamp: "Formatted at 2026-09-29 20:27:45 on dist-test-slave-zbnc"
server_key: "b41ed46d4a46c8b4ae292ac1ce9aac39"
server_key_iv: "e9782158bd70c990298cfaf5d7e6837a"
server_key_version: "encryptionkey@0"
I20260929 20:27:45.090551 5726 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790713339161492-17621-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790713339161492-17621-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790713339161492-17621-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:27:45.124563 5726 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:27:45.128839 5726 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:27:45.129146 5726 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:27:45.149405 5726 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33539
I20260929 20:27:45.149428 5792 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33539 every 8 connection(s)
I20260929 20:27:45.150848 5726 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790713339161492-17621-0/minicluster-data/master-0/data/info.pb
I20260929 20:27:45.154667 5793 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:27:45.157083 17621 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu as pid 5726
I20260929 20:27:45.157250 17621 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790713339161492-17621-0/minicluster-data/master-0/wal/instance
I20260929 20:27:45.157675 17621 external_mini_cluster.cc:1428] Setting key 9e34fe47606ce29e840300ebe4b08613
I20260929 20:27:45.166177 5793 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5ffad14b1f2d40f8a08bde8d66b1a0af: Bootstrap starting.
I20260929 20:27:45.169863 5793 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5ffad14b1f2d40f8a08bde8d66b1a0af: Neither blocks nor log segments found. Creating new log.
I20260929 20:27:45.170692 5796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:27:45.159489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60181 (local address 127.0.0.1:33539)
0929 20:27:45.160246 (+ 757us) server_negotiation.cc:206] Beginning negotiation
0929 20:27:45.160258 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 20:27:45.160297 (+ 39us) server_negotiation.cc:407] Connection header received
0929 20:27:45.162030 (+ 1733us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:27:45.162054 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:27:45.162449 (+ 395us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:27:45.162824 (+ 375us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:27:45.163404 (+ 580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:27:45.164740 (+ 1336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:27:45.166052 (+ 1312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:27:45.166831 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:27:45.167020 (+ 189us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:27:45.168370 (+ 1350us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:27:45.168378 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:27:45.168396 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:27:45.168910 (+ 514us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:27:45.168983 (+ 73us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:27:45.169007 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:27:45.169134 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0929 20:27:45.169463 (+ 329us) server_negotiation.cc:299] Negotiation successful
0929 20:27:45.169648 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":547,"thread_start_us":326,"threads_started":1}
I20260929 20:27:45.171036 5793 log.cc:824] T 00000000000000000000000000000000 P 5ffad14b1f2d40f8a08bde8d66b1a0af: Log is configured to *not* fsync() on all Append() calls
I20260929 20:27:45.174374 5793 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5ffad14b1f2d40f8a08bde8d66b1a0af: No bootstrap required, opened a new log
I20260929 20:27:45.178952 17621 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 20:27:45.184267 5793 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5ffad14b1f2d40f8a08bde8d66b1a0af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ffad14b1f2d40f8a08bde8d66b1a0af" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33539 } }
I20260929 20:27:45.184847 5793 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5ffad14b1f2d40f8a08bde8d66b1a0af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:27:45.185003 5793 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5ffad14b1f2d40f8a08bde8d66b1a0af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ffad14b1f2d40f8a08bde8d66b1a0af, State: Initialized, Role: FOLLOWER
I20260929 20:27:45.185781 5793 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5ffad14b1f2d40f8a08bde8d66b1a0af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ffad14b1f2d40f8a08bde8d66b1a0af" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33539 } }
I20260929 20:27:45.186141 5793 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5ffad14b1f2d40f8a08bde8d66b1a0af [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 20:27:45.186305 5793 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5ffad14b1f2d40f8a08bde8d66b1a0af [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 20:27:45.186507 5793 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5ffad14b1f2d40f8a08bde8d66b1a0af [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:27:45.190377 5793 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5ffad14b1f2d40f8a08bde8d66b1a0af [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ffad14b1f2d40f8a08bde8d66b1a0af" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33539 } }
I20260929 20:27:45.191165 5793 leader_election.cc:302] T 00000000000000000000000000000000 P 5ffad14b1f2d40f8a08bde8d66b1a0af [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5ffad14b1f2d40f8a08bde8d66b1a0af; no voters:
I20260929 20:27:45.192171 5793 leader_election.cc:288] T 00000000000000000000000000000000 P 5ffad14b1f2d40f8a08bde8d66b1a0af [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 20:27:45.192310 5803 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5ffad14b1f2d40f8a08bde8d66b1a0af [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:27:45.194177 5803 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5ffad14b1f2d40f8a08bde8d66b1a0af [term 1 LEADER]: Becoming Leader. State: Replica: 5ffad14b1f2d40f8a08bde8d66b1a0af, State: Running, Role: LEADER
I20260929 20:27:45.194468 5793 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5ffad14b1f2d40f8a08bde8d66b1a0af [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:27:45.194906 5803 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5ffad14b1f2d40f8a08bde8d66b1a0af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ffad14b1f2d40f8a08bde8d66b1a0af" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33539 } }
I20260929 20:27:45.195493 5796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:27:45.184862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35731 (local address 127.0.0.1:33539)
0929 20:27:45.185099 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0929 20:27:45.185107 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 20:27:45.187583 (+ 2476us) server_negotiation.cc:407] Connection header received
0929 20:27:45.187807 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:27:45.187816 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:27:45.187919 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:27:45.188048 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:27:45.188604 (+ 556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:27:45.189972 (+ 1368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:27:45.191815 (+ 1843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:27:45.192623 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:27:45.192768 (+ 145us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:27:45.194531 (+ 1763us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:27:45.194541 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:27:45.194548 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:27:45.194584 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:27:45.194678 (+ 94us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:27:45.194700 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:27:45.194830 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0929 20:27:45.195100 (+ 270us) server_negotiation.cc:299] Negotiation successful
0929 20:27:45.195163 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260929 20:27:45.199476 5805 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5ffad14b1f2d40f8a08bde8d66b1a0af [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5ffad14b1f2d40f8a08bde8d66b1a0af. Latest consensus state: current_term: 1 leader_uuid: "5ffad14b1f2d40f8a08bde8d66b1a0af" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ffad14b1f2d40f8a08bde8d66b1a0af" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33539 } } }
I20260929 20:27:45.199448 5804 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5ffad14b1f2d40f8a08bde8d66b1a0af [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5ffad14b1f2d40f8a08bde8d66b1a0af" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5ffad14b1f2d40f8a08bde8d66b1a0af" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33539 } } }
I20260929 20:27:45.199985 5804 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5ffad14b1f2d40f8a08bde8d66b1a0af [sys.catalog]: This master's current role is: LEADER
I20260929 20:27:45.199985 5805 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5ffad14b1f2d40f8a08bde8d66b1a0af [sys.catalog]: This master's current role is: LEADER
I20260929 20:27:45.200800 5814 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:27:45.206238 5814 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:27:45.215763 5814 catalog_manager.cc:1347] Generated new cluster ID: 2e71404691174578b5aa987b0c1034e2
I20260929 20:27:45.215901 5814 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:27:45.234323 5814 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 20:27:45.236068 5814 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:27:45.253000 5814 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5ffad14b1f2d40f8a08bde8d66b1a0af: Generated new TSK 0
I20260929 20:27:45.254225 5814 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 20:27:45.269320 17621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu pid 5726
W20260929 20:27:45.270735 5788 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:60181
W20260929 20:27:45.271497 17621 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu pid 5726: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 20:27:45.271692 17621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu with pid 5726
W20260929 20:27:45.289004 5711 connection.cc:535] client connection to 127.0.0.1:33539 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:33539): Connection reset by peer (error 104)
2026-09-29T20:27:45Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1964 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790713339161492-17621-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 20:27:45 dist-test-slave-zbnc krb5kdc[5824](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 20:27:45 dist-test-slave-zbnc krb5kdc[5824](info): set up 2 sockets
Sep 29 20:27:45 dist-test-slave-zbnc krb5kdc[5824](info): commencing operation
krb5kdc: starting...
W20260929 20:27:46.404627 17621 mini_kdc.cc:121] Time spent starting KDC: real 1.096s user 0.004s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790713339161492-17621-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790713339161492-17621-0/krb5kdc/test-user.keytab.
Sep 29 20:27:46 dist-test-slave-zbnc krb5kdc[5824](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713666, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T20:27:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T20:27:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790713339161492-17621-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790713339161492-17621-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 20:27:46.671545 17621 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
/tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790713339161492-17621-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790713339161492-17621-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790713339161492-17621-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790713339161492-17621-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40329
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790713339161492-17621-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41849
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40329
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790713339161492-17621-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790713339161492-17621-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790713339161492-17621-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:27:47.010210 5837 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:27:47.010560 5837 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:27:47.010735 5837 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:27:47.018215 5837 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:27:47.018381 5837 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:27:47.018491 5837 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:27:47.018599 5837 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:27:47.018683 5837 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:27:47.029141 5837 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41849
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790713339161492-17621-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790713339161492-17621-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40329
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790713339161492-17621-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40329
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790713339161492-17621-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790713339161492-17621-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:27:47.030529 5837 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:27:47.031968 5837 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:27:47.044358 5845 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:27:47.044320 5842 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:27:47.044389 5843 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:27:47.045125 5837 server_base.cc:1034] running on GCE node
I20260929 20:27:47.046056 5837 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:27:47.047551 5837 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:27:47.048787 5837 hybrid_clock.cc:648] HybridClock initialized: now 1790713667048732 us; error 41 us; skew 500 ppm
I20260929 20:27:47.049201 5837 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:27:47 dist-test-slave-zbnc krb5kdc[5824](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713667, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:27:47.056531 5837 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 20:27:47.059705 5837 webserver.cc:460] Webserver started at http://127.0.0.1:46567/ using document root <none> and password file <none>
I20260929 20:27:47.060410 5837 fs_manager.cc:359] Metadata directory not provided
I20260929 20:27:47.060513 5837 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:27:47.060762 5837 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:27:47.063843 5837 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790713339161492-17621-0/minicluster-data/master-0/data/instance:
uuid: "41810ff78be04507a99232bf6a7768bd"
format_stamp: "Formatted at 2026-09-29 20:27:47 on dist-test-slave-zbnc"
server_key: "cfd41b4e7e8475fdcb924dc2fa88ba07"
server_key_iv: "d83496d1960a7c1b1430338cbb19bf7f"
server_key_version: "encryptionkey@0"
I20260929 20:27:47.064567 5837 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790713339161492-17621-0/minicluster-data/master-0/wal/instance:
uuid: "41810ff78be04507a99232bf6a7768bd"
format_stamp: "Formatted at 2026-09-29 20:27:47 on dist-test-slave-zbnc"
server_key: "cfd41b4e7e8475fdcb924dc2fa88ba07"
server_key_iv: "d83496d1960a7c1b1430338cbb19bf7f"
server_key_version: "encryptionkey@0"
I20260929 20:27:47.069895 5837 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 20:27:47.073534 5852 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:27:47.075027 5837 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 20:27:47.075193 5837 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790713339161492-17621-0/minicluster-data/master-0/data,/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790713339161492-17621-0/minicluster-data/master-0/wal
uuid: "41810ff78be04507a99232bf6a7768bd"
format_stamp: "Formatted at 2026-09-29 20:27:47 on dist-test-slave-zbnc"
server_key: "cfd41b4e7e8475fdcb924dc2fa88ba07"
server_key_iv: "d83496d1960a7c1b1430338cbb19bf7f"
server_key_version: "encryptionkey@0"
I20260929 20:27:47.075462 5837 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790713339161492-17621-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790713339161492-17621-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790713339161492-17621-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:27:47.081332 5837 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:27:47.085709 5837 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:27:47.085999 5837 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:27:47.104967 5837 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40329
I20260929 20:27:47.105008 5903 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40329 every 8 connection(s)
I20260929 20:27:47.106479 5837 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790713339161492-17621-0/minicluster-data/master-0/data/info.pb
I20260929 20:27:47.110307 5904 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:27:47.116195 17621 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu as pid 5837
I20260929 20:27:47.116410 17621 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790713339161492-17621-0/minicluster-data/master-0/wal/instance
I20260929 20:27:47.116906 17621 external_mini_cluster.cc:1428] Setting key e5fe316454ae5fd7e1b867e8d0a2902d
I20260929 20:27:47.122231 5904 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 41810ff78be04507a99232bf6a7768bd: Bootstrap starting.
I20260929 20:27:47.126145 5907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:27:47.118831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51019 (local address 127.0.0.1:40329)
0929 20:27:47.119755 (+ 924us) server_negotiation.cc:206] Beginning negotiation
0929 20:27:47.119774 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0929 20:27:47.119828 (+ 54us) server_negotiation.cc:407] Connection header received
0929 20:27:47.122247 (+ 2419us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:27:47.122277 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:27:47.122762 (+ 485us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:27:47.123204 (+ 442us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:27:47.124213 (+ 1009us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:27:47.124222 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:27:47.124241 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:27:47.124258 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:27:47.124347 (+ 89us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:27:47.124380 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:27:47.124590 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0929 20:27:47.124722 (+ 132us) server_negotiation.cc:299] Negotiation successful
0929 20:27:47.124911 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":623,"thread_start_us":384,"threads_started":1}
I20260929 20:27:47.126434 5904 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 41810ff78be04507a99232bf6a7768bd: Neither blocks nor log segments found. Creating new log.
I20260929 20:27:47.127619 5904 log.cc:824] T 00000000000000000000000000000000 P 41810ff78be04507a99232bf6a7768bd: Log is configured to *not* fsync() on all Append() calls
I20260929 20:27:47.130668 17621 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 20:27:47.130968 5904 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 41810ff78be04507a99232bf6a7768bd: No bootstrap required, opened a new log
I20260929 20:27:47.138445 5907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:27:47.135894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52683 (local address 127.0.0.1:40329)
0929 20:27:47.136123 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0929 20:27:47.136130 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:27:47.136496 (+ 366us) server_negotiation.cc:407] Connection header received
0929 20:27:47.136674 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:27:47.136677 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:27:47.136726 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:27:47.136811 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:27:47.137745 (+ 934us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:27:47.137748 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:27:47.137751 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:27:47.137765 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:27:47.137809 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:27:47.137829 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:27:47.137939 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0929 20:27:47.138059 (+ 120us) server_negotiation.cc:299] Negotiation successful
0929 20:27:47.138105 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260929 20:27:47.141377 5904 raft_consensus.cc:348] T 00000000000000000000000000000000 P 41810ff78be04507a99232bf6a7768bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41810ff78be04507a99232bf6a7768bd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40329 } }
I20260929 20:27:47.141762 5904 raft_consensus.cc:374] T 00000000000000000000000000000000 P 41810ff78be04507a99232bf6a7768bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:27:47.141862 5904 raft_consensus.cc:729] T 00000000000000000000000000000000 P 41810ff78be04507a99232bf6a7768bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41810ff78be04507a99232bf6a7768bd, State: Initialized, Role: FOLLOWER
I20260929 20:27:47.142789 5904 consensus_queue.cc:260] T 00000000000000000000000000000000 P 41810ff78be04507a99232bf6a7768bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41810ff78be04507a99232bf6a7768bd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40329 } }
I20260929 20:27:47.143153 5904 raft_consensus.cc:388] T 00000000000000000000000000000000 P 41810ff78be04507a99232bf6a7768bd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 20:27:47.143335 5904 raft_consensus.cc:482] T 00000000000000000000000000000000 P 41810ff78be04507a99232bf6a7768bd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 20:27:47.143527 5904 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 41810ff78be04507a99232bf6a7768bd [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:27:47.146195 5904 raft_consensus.cc:504] T 00000000000000000000000000000000 P 41810ff78be04507a99232bf6a7768bd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41810ff78be04507a99232bf6a7768bd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40329 } }
I20260929 20:27:47.146682 5904 leader_election.cc:302] T 00000000000000000000000000000000 P 41810ff78be04507a99232bf6a7768bd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 41810ff78be04507a99232bf6a7768bd; no voters:
I20260929 20:27:47.147339 5904 leader_election.cc:288] T 00000000000000000000000000000000 P 41810ff78be04507a99232bf6a7768bd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 20:27:47.147511 5914 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 41810ff78be04507a99232bf6a7768bd [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:27:47.148290 5914 raft_consensus.cc:686] T 00000000000000000000000000000000 P 41810ff78be04507a99232bf6a7768bd [term 1 LEADER]: Becoming Leader. State: Replica: 41810ff78be04507a99232bf6a7768bd, State: Running, Role: LEADER
I20260929 20:27:47.148854 5914 consensus_queue.cc:237] T 00000000000000000000000000000000 P 41810ff78be04507a99232bf6a7768bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41810ff78be04507a99232bf6a7768bd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40329 } }
I20260929 20:27:47.149737 5904 sys_catalog.cc:564] T 00000000000000000000000000000000 P 41810ff78be04507a99232bf6a7768bd [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:27:47.151937 5915 sys_catalog.cc:455] T 00000000000000000000000000000000 P 41810ff78be04507a99232bf6a7768bd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "41810ff78be04507a99232bf6a7768bd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41810ff78be04507a99232bf6a7768bd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40329 } } }
I20260929 20:27:47.152385 5915 sys_catalog.cc:458] T 00000000000000000000000000000000 P 41810ff78be04507a99232bf6a7768bd [sys.catalog]: This master's current role is: LEADER
I20260929 20:27:47.152580 5916 sys_catalog.cc:455] T 00000000000000000000000000000000 P 41810ff78be04507a99232bf6a7768bd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 41810ff78be04507a99232bf6a7768bd. Latest consensus state: current_term: 1 leader_uuid: "41810ff78be04507a99232bf6a7768bd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41810ff78be04507a99232bf6a7768bd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40329 } } }
I20260929 20:27:47.153110 5916 sys_catalog.cc:458] T 00000000000000000000000000000000 P 41810ff78be04507a99232bf6a7768bd [sys.catalog]: This master's current role is: LEADER
I20260929 20:27:47.155123 5920 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:27:47.160614 5920 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:27:47.172781 5920 catalog_manager.cc:1347] Generated new cluster ID: d36475c9753c40aeb03079e4b2ee66c2
I20260929 20:27:47.172914 5920 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:27:47.195142 5920 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 20:27:47.195750 5920 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:27:47.208787 5920 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 41810ff78be04507a99232bf6a7768bd: Generated new TSK 0
I20260929 20:27:47.209895 5920 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 20:27:47.224478 17621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu pid 5837
W20260929 20:27:47.225674 5899 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:51019
W20260929 20:27:47.226462 17621 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu pid 5837: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 20:27:47.226647 17621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu with pid 5837
2026-09-29T20:27:47Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1954 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790713339161492-17621-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 20:27:47 dist-test-slave-zbnc krb5kdc[5935](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 20:27:47 dist-test-slave-zbnc krb5kdc[5935](info): set up 2 sockets
Sep 29 20:27:47 dist-test-slave-zbnc krb5kdc[5935](info): commencing operation
krb5kdc: starting...
W20260929 20:27:48.340827 17621 mini_kdc.cc:121] Time spent starting KDC: real 1.077s user 0.000s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790713339161492-17621-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790713339161492-17621-0/krb5kdc/test-user.keytab.
Sep 29 20:27:48 dist-test-slave-zbnc krb5kdc[5935](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713668, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T20:27:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T20:27:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790713339161492-17621-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790713339161492-17621-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 20:27:48.602519 17621 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
/tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790713339161492-17621-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790713339161492-17621-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790713339161492-17621-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790713339161492-17621-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45859
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790713339161492-17621-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40417
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45859
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790713339161492-17621-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790713339161492-17621-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790713339161492-17621-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:27:48.935091 5948 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:27:48.935401 5948 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:27:48.935495 5948 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:27:48.942597 5948 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:27:48.942710 5948 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:27:48.942761 5948 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:27:48.942823 5948 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:27:48.942911 5948 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:27:48.952867 5948 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40417
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790713339161492-17621-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790713339161492-17621-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45859
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790713339161492-17621-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45859
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790713339161492-17621-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790713339161492-17621-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:27:48.954130 5948 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:27:48.955447 5948 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:27:48.967469 5956 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:27:48.967428 5953 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:27:48.967507 5954 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:27:48.968480 5948 server_base.cc:1034] running on GCE node
I20260929 20:27:48.969254 5948 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:27:48.970753 5948 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:27:48.971957 5948 hybrid_clock.cc:648] HybridClock initialized: now 1790713668971918 us; error 38 us; skew 500 ppm
I20260929 20:27:48.972381 5948 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:27:48 dist-test-slave-zbnc krb5kdc[5935](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713668, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:27:48.975663 5948 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 20:27:48.977116 5948 webserver.cc:460] Webserver started at http://127.0.0.1:34415/ using document root <none> and password file <none>
I20260929 20:27:48.977756 5948 fs_manager.cc:359] Metadata directory not provided
I20260929 20:27:48.977864 5948 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:27:48.978111 5948 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:27:48.981119 5948 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790713339161492-17621-0/minicluster-data/master-0/data/instance:
uuid: "31d9da11eb1242f9bf158cf5b55d1b63"
format_stamp: "Formatted at 2026-09-29 20:27:48 on dist-test-slave-zbnc"
server_key: "556402f6d2ec3d531b101dc301779d38"
server_key_iv: "aadc0d2382ea40b32e211805a36e8adc"
server_key_version: "encryptionkey@0"
I20260929 20:27:48.982023 5948 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790713339161492-17621-0/minicluster-data/master-0/wal/instance:
uuid: "31d9da11eb1242f9bf158cf5b55d1b63"
format_stamp: "Formatted at 2026-09-29 20:27:48 on dist-test-slave-zbnc"
server_key: "556402f6d2ec3d531b101dc301779d38"
server_key_iv: "aadc0d2382ea40b32e211805a36e8adc"
server_key_version: "encryptionkey@0"
I20260929 20:27:48.987257 5948 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260929 20:27:48.990687 5963 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:27:48.992095 5948 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 20:27:48.992271 5948 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790713339161492-17621-0/minicluster-data/master-0/data,/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790713339161492-17621-0/minicluster-data/master-0/wal
uuid: "31d9da11eb1242f9bf158cf5b55d1b63"
format_stamp: "Formatted at 2026-09-29 20:27:48 on dist-test-slave-zbnc"
server_key: "556402f6d2ec3d531b101dc301779d38"
server_key_iv: "aadc0d2382ea40b32e211805a36e8adc"
server_key_version: "encryptionkey@0"
I20260929 20:27:48.992471 5948 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790713339161492-17621-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790713339161492-17621-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790713339161492-17621-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:27:49.010013 5948 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:27:49.014164 5948 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:27:49.014460 5948 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:27:49.033514 5948 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45859
I20260929 20:27:49.033540 6014 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45859 every 8 connection(s)
I20260929 20:27:49.035012 5948 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790713339161492-17621-0/minicluster-data/master-0/data/info.pb
I20260929 20:27:49.037907 17621 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu as pid 5948
I20260929 20:27:49.038123 17621 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790713339161492-17621-0/minicluster-data/master-0/wal/instance
I20260929 20:27:49.038686 17621 external_mini_cluster.cc:1428] Setting key 7f4e28dcf8c61779313a37e92b5db712
I20260929 20:27:49.039296 6015 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 29 20:27:49 dist-test-slave-zbnc krb5kdc[5935](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713668, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260929 20:27:49.053197 6015 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 31d9da11eb1242f9bf158cf5b55d1b63: Bootstrap starting.
I20260929 20:27:49.057760 6015 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 31d9da11eb1242f9bf158cf5b55d1b63: Neither blocks nor log segments found. Creating new log.
I20260929 20:27:49.059083 6015 log.cc:824] T 00000000000000000000000000000000 P 31d9da11eb1242f9bf158cf5b55d1b63: Log is configured to *not* fsync() on all Append() calls
I20260929 20:27:49.063617 6015 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 31d9da11eb1242f9bf158cf5b55d1b63: No bootstrap required, opened a new log
I20260929 20:27:49.064318 6018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:27:49.040625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40795 (local address 127.0.0.1:45859)
0929 20:27:49.041370 (+ 745us) server_negotiation.cc:206] Beginning negotiation
0929 20:27:49.041383 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 20:27:49.041424 (+ 41us) server_negotiation.cc:407] Connection header received
0929 20:27:49.043039 (+ 1615us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:27:49.043061 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:27:49.043485 (+ 424us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:27:49.043886 (+ 401us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:27:49.045402 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:27:49.046938 (+ 1536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:27:49.048163 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:27:49.048866 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:27:49.049092 (+ 226us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:27:49.054584 (+ 5492us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:27:49.054598 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:27:49.054620 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:27:49.054677 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:27:49.058961 (+ 4284us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:27:49.059873 (+ 912us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:27:49.059882 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:27:49.059890 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:27:49.060140 (+ 250us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:27:49.060737 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:27:49.060744 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:27:49.060749 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:27:49.061536 (+ 787us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:27:49.061849 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0929 20:27:49.062838 (+ 989us) server_negotiation.cc:299] Negotiation successful
0929 20:27:49.063228 (+ 390us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":540,"thread_start_us":328,"threads_started":1}
I20260929 20:27:49.068581 17621 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 20:27:49.072431 6015 raft_consensus.cc:348] T 00000000000000000000000000000000 P 31d9da11eb1242f9bf158cf5b55d1b63 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31d9da11eb1242f9bf158cf5b55d1b63" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45859 } }
I20260929 20:27:49.073159 6015 raft_consensus.cc:374] T 00000000000000000000000000000000 P 31d9da11eb1242f9bf158cf5b55d1b63 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:27:49.073298 6015 raft_consensus.cc:729] T 00000000000000000000000000000000 P 31d9da11eb1242f9bf158cf5b55d1b63 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31d9da11eb1242f9bf158cf5b55d1b63, State: Initialized, Role: FOLLOWER
I20260929 20:27:49.074353 6015 consensus_queue.cc:260] T 00000000000000000000000000000000 P 31d9da11eb1242f9bf158cf5b55d1b63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31d9da11eb1242f9bf158cf5b55d1b63" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45859 } }
I20260929 20:27:49.074724 6015 raft_consensus.cc:388] T 00000000000000000000000000000000 P 31d9da11eb1242f9bf158cf5b55d1b63 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 20:27:49.074950 6015 raft_consensus.cc:482] T 00000000000000000000000000000000 P 31d9da11eb1242f9bf158cf5b55d1b63 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 20:27:49.075196 6015 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 31d9da11eb1242f9bf158cf5b55d1b63 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:27:49.077456 6015 raft_consensus.cc:504] T 00000000000000000000000000000000 P 31d9da11eb1242f9bf158cf5b55d1b63 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31d9da11eb1242f9bf158cf5b55d1b63" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45859 } }
I20260929 20:27:49.077996 6015 leader_election.cc:302] T 00000000000000000000000000000000 P 31d9da11eb1242f9bf158cf5b55d1b63 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 31d9da11eb1242f9bf158cf5b55d1b63; no voters:
I20260929 20:27:49.078680 6015 leader_election.cc:288] T 00000000000000000000000000000000 P 31d9da11eb1242f9bf158cf5b55d1b63 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 20:27:49.078771 6025 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 31d9da11eb1242f9bf158cf5b55d1b63 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:27:49.081224 6025 raft_consensus.cc:686] T 00000000000000000000000000000000 P 31d9da11eb1242f9bf158cf5b55d1b63 [term 1 LEADER]: Becoming Leader. State: Replica: 31d9da11eb1242f9bf158cf5b55d1b63, State: Running, Role: LEADER
I20260929 20:27:49.081440 6015 sys_catalog.cc:564] T 00000000000000000000000000000000 P 31d9da11eb1242f9bf158cf5b55d1b63 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:27:49.081782 6025 consensus_queue.cc:237] T 00000000000000000000000000000000 P 31d9da11eb1242f9bf158cf5b55d1b63 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31d9da11eb1242f9bf158cf5b55d1b63" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45859 } }
I20260929 20:27:49.085412 6027 sys_catalog.cc:455] T 00000000000000000000000000000000 P 31d9da11eb1242f9bf158cf5b55d1b63 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "31d9da11eb1242f9bf158cf5b55d1b63" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31d9da11eb1242f9bf158cf5b55d1b63" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45859 } } }
I20260929 20:27:49.086079 6027 sys_catalog.cc:458] T 00000000000000000000000000000000 P 31d9da11eb1242f9bf158cf5b55d1b63 [sys.catalog]: This master's current role is: LEADER
I20260929 20:27:49.085911 6026 sys_catalog.cc:455] T 00000000000000000000000000000000 P 31d9da11eb1242f9bf158cf5b55d1b63 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 31d9da11eb1242f9bf158cf5b55d1b63. Latest consensus state: current_term: 1 leader_uuid: "31d9da11eb1242f9bf158cf5b55d1b63" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31d9da11eb1242f9bf158cf5b55d1b63" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45859 } } }
I20260929 20:27:49.086352 6026 sys_catalog.cc:458] T 00000000000000000000000000000000 P 31d9da11eb1242f9bf158cf5b55d1b63 [sys.catalog]: This master's current role is: LEADER
I20260929 20:27:49.091089 6018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:27:49.072924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37307 (local address 127.0.0.1:45859)
0929 20:27:49.073120 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0929 20:27:49.073126 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:27:49.075410 (+ 2284us) server_negotiation.cc:407] Connection header received
0929 20:27:49.075615 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:27:49.075621 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:27:49.075694 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:27:49.075831 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:27:49.077232 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:27:49.078035 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:27:49.079608 (+ 1573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:27:49.080155 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:27:49.080228 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:27:49.083162 (+ 2934us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:27:49.083169 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:27:49.083176 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:27:49.083224 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:27:49.088207 (+ 4983us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:27:49.088946 (+ 739us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:27:49.088951 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:27:49.088955 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:27:49.089137 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:27:49.089519 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:27:49.089524 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:27:49.089528 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:27:49.089825 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:27:49.090103 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0929 20:27:49.090443 (+ 340us) server_negotiation.cc:299] Negotiation successful
0929 20:27:49.090665 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260929 20:27:49.092760 6034 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:27:49.097815 6034 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:27:49.107429 6034 catalog_manager.cc:1347] Generated new cluster ID: 519dd1dde6bd45ce81114b05457f364d
I20260929 20:27:49.107606 6034 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:27:49.125870 6034 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 20:27:49.127386 6034 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:27:49.140754 6034 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 31d9da11eb1242f9bf158cf5b55d1b63: Generated new TSK 0
I20260929 20:27:49.141731 6034 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 20:27:49.157752 17621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu pid 5948
I20260929 20:27:49.159052 6010 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:27:49.255024 17621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu with pid 5948
2026-09-29T20:27:49Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (2029 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790713339161492-17621-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 20:27:49 dist-test-slave-zbnc krb5kdc[6047](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 20:27:49 dist-test-slave-zbnc krb5kdc[6047](info): set up 2 sockets
Sep 29 20:27:49 dist-test-slave-zbnc krb5kdc[6047](info): commencing operation
krb5kdc: starting...
W20260929 20:27:50.368674 17621 mini_kdc.cc:121] Time spent starting KDC: real 1.076s user 0.000s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790713339161492-17621-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790713339161492-17621-0/krb5kdc/test-user.keytab.
Sep 29 20:27:50 dist-test-slave-zbnc krb5kdc[6047](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713670, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T20:27:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T20:27:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790713339161492-17621-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790713339161492-17621-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 20:27:50.635978 17621 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
/tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790713339161492-17621-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790713339161492-17621-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790713339161492-17621-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790713339161492-17621-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32769
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790713339161492-17621-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37879
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:32769
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790713339161492-17621-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790713339161492-17621-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790713339161492-17621-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:27:50.969318 6060 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:27:50.969597 6060 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:27:50.969785 6060 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:27:50.977038 6060 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:27:50.977140 6060 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:27:50.977246 6060 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:27:50.977330 6060 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:27:50.977421 6060 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:27:50.987315 6060 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37879
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790713339161492-17621-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790713339161492-17621-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:32769
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790713339161492-17621-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32769
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790713339161492-17621-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790713339161492-17621-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:27:50.988575 6060 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:27:50.989907 6060 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:27:51.002028 6068 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:27:51.001945 6066 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:27:51.001974 6065 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:27:51.003033 6060 server_base.cc:1034] running on GCE node
I20260929 20:27:51.003795 6060 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:27:51.005237 6060 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:27:51.006470 6060 hybrid_clock.cc:648] HybridClock initialized: now 1790713671006421 us; error 47 us; skew 500 ppm
I20260929 20:27:51.006826 6060 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:27:51 dist-test-slave-zbnc krb5kdc[6047](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713671, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:27:51.010016 6060 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 20:27:51.011534 6060 webserver.cc:460] Webserver started at http://127.0.0.1:44421/ using document root <none> and password file <none>
I20260929 20:27:51.012188 6060 fs_manager.cc:359] Metadata directory not provided
I20260929 20:27:51.012321 6060 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:27:51.012646 6060 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:27:51.015822 6060 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790713339161492-17621-0/minicluster-data/master-0/data/instance:
uuid: "6fd39200796944bab2bd9c348d191aa9"
format_stamp: "Formatted at 2026-09-29 20:27:51 on dist-test-slave-zbnc"
server_key: "f5df74a8e733071677f9ae6878a36ec8"
server_key_iv: "38da7a20ae4d492939cb4f7962c34a3b"
server_key_version: "encryptionkey@0"
I20260929 20:27:51.016571 6060 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790713339161492-17621-0/minicluster-data/master-0/wal/instance:
uuid: "6fd39200796944bab2bd9c348d191aa9"
format_stamp: "Formatted at 2026-09-29 20:27:51 on dist-test-slave-zbnc"
server_key: "f5df74a8e733071677f9ae6878a36ec8"
server_key_iv: "38da7a20ae4d492939cb4f7962c34a3b"
server_key_version: "encryptionkey@0"
I20260929 20:27:51.021593 6060 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 20:27:51.025182 6075 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:27:51.026657 6060 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260929 20:27:51.026895 6060 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790713339161492-17621-0/minicluster-data/master-0/data,/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790713339161492-17621-0/minicluster-data/master-0/wal
uuid: "6fd39200796944bab2bd9c348d191aa9"
format_stamp: "Formatted at 2026-09-29 20:27:51 on dist-test-slave-zbnc"
server_key: "f5df74a8e733071677f9ae6878a36ec8"
server_key_iv: "38da7a20ae4d492939cb4f7962c34a3b"
server_key_version: "encryptionkey@0"
I20260929 20:27:51.027086 6060 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790713339161492-17621-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790713339161492-17621-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790713339161492-17621-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:27:51.048439 6060 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:27:51.053467 6060 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:27:51.053850 6060 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:27:51.076629 6060 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:32769
I20260929 20:27:51.076654 6126 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:32769 every 8 connection(s)
I20260929 20:27:51.078292 6060 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790713339161492-17621-0/minicluster-data/master-0/data/info.pb
I20260929 20:27:51.081813 17621 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu as pid 6060
I20260929 20:27:51.082039 17621 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790713339161492-17621-0/minicluster-data/master-0/wal/instance
I20260929 20:27:51.082633 17621 external_mini_cluster.cc:1428] Setting key dff55e82cd192d3c5dd38442528944e2
I20260929 20:27:51.083164 6127 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:27:51.099167 6127 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6fd39200796944bab2bd9c348d191aa9: Bootstrap starting.
I20260929 20:27:51.099349 6130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:27:51.084918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58499 (local address 127.0.0.1:32769)
0929 20:27:51.085941 (+ 1023us) server_negotiation.cc:206] Beginning negotiation
0929 20:27:51.085959 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0929 20:27:51.086318 (+ 359us) server_negotiation.cc:407] Connection header received
0929 20:27:51.088348 (+ 2030us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:27:51.088376 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:27:51.088967 (+ 591us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:27:51.089479 (+ 512us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:27:51.090049 (+ 570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:27:51.092188 (+ 2139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:27:51.093649 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:27:51.094801 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:27:51.095052 (+ 251us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:27:51.096270 (+ 1218us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:27:51.096281 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:27:51.096300 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:27:51.097126 (+ 826us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:27:51.097247 (+ 121us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:27:51.097281 (+ 34us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:27:51.097387 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0929 20:27:51.097652 (+ 265us) server_negotiation.cc:299] Negotiation successful
0929 20:27:51.097946 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":738,"thread_start_us":432,"threads_started":1}
I20260929 20:27:51.102710 6127 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6fd39200796944bab2bd9c348d191aa9: Neither blocks nor log segments found. Creating new log.
I20260929 20:27:51.102972 17621 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 20:27:51.104084 6127 log.cc:824] T 00000000000000000000000000000000 P 6fd39200796944bab2bd9c348d191aa9: Log is configured to *not* fsync() on all Append() calls
I20260929 20:27:51.107594 6127 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6fd39200796944bab2bd9c348d191aa9: No bootstrap required, opened a new log
I20260929 20:27:51.115607 6130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:27:51.108535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45419 (local address 127.0.0.1:32769)
0929 20:27:51.108739 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0929 20:27:51.108746 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:27:51.109200 (+ 454us) server_negotiation.cc:407] Connection header received
0929 20:27:51.109352 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:27:51.109363 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:27:51.109458 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:27:51.109590 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:27:51.110078 (+ 488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:27:51.111275 (+ 1197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:27:51.112618 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:27:51.113448 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:27:51.113606 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:27:51.114668 (+ 1062us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:27:51.114675 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:27:51.114681 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:27:51.114714 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:27:51.114769 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:27:51.114792 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:27:51.114928 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0929 20:27:51.115074 (+ 146us) server_negotiation.cc:299] Negotiation successful
0929 20:27:51.115192 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260929 20:27:51.118297 6127 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6fd39200796944bab2bd9c348d191aa9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fd39200796944bab2bd9c348d191aa9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32769 } }
I20260929 20:27:51.118706 6127 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6fd39200796944bab2bd9c348d191aa9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:27:51.118794 6127 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6fd39200796944bab2bd9c348d191aa9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6fd39200796944bab2bd9c348d191aa9, State: Initialized, Role: FOLLOWER
I20260929 20:27:51.119598 6127 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6fd39200796944bab2bd9c348d191aa9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fd39200796944bab2bd9c348d191aa9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32769 } }
I20260929 20:27:51.120028 6127 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6fd39200796944bab2bd9c348d191aa9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 20:27:51.120239 6127 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6fd39200796944bab2bd9c348d191aa9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 20:27:51.120464 6127 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6fd39200796944bab2bd9c348d191aa9 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:27:51.123189 6127 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6fd39200796944bab2bd9c348d191aa9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fd39200796944bab2bd9c348d191aa9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32769 } }
I20260929 20:27:51.123827 6127 leader_election.cc:302] T 00000000000000000000000000000000 P 6fd39200796944bab2bd9c348d191aa9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6fd39200796944bab2bd9c348d191aa9; no voters:
I20260929 20:27:51.124424 6127 leader_election.cc:288] T 00000000000000000000000000000000 P 6fd39200796944bab2bd9c348d191aa9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 20:27:51.124593 6137 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6fd39200796944bab2bd9c348d191aa9 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:27:51.125393 6137 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6fd39200796944bab2bd9c348d191aa9 [term 1 LEADER]: Becoming Leader. State: Replica: 6fd39200796944bab2bd9c348d191aa9, State: Running, Role: LEADER
I20260929 20:27:51.125972 6137 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6fd39200796944bab2bd9c348d191aa9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fd39200796944bab2bd9c348d191aa9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32769 } }
I20260929 20:27:51.126652 6127 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6fd39200796944bab2bd9c348d191aa9 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:27:51.128901 6138 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6fd39200796944bab2bd9c348d191aa9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6fd39200796944bab2bd9c348d191aa9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fd39200796944bab2bd9c348d191aa9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32769 } } }
I20260929 20:27:51.129380 6138 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6fd39200796944bab2bd9c348d191aa9 [sys.catalog]: This master's current role is: LEADER
I20260929 20:27:51.130676 6145 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:27:51.130702 6139 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6fd39200796944bab2bd9c348d191aa9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6fd39200796944bab2bd9c348d191aa9. Latest consensus state: current_term: 1 leader_uuid: "6fd39200796944bab2bd9c348d191aa9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fd39200796944bab2bd9c348d191aa9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32769 } } }
I20260929 20:27:51.131110 6139 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6fd39200796944bab2bd9c348d191aa9 [sys.catalog]: This master's current role is: LEADER
I20260929 20:27:51.137693 6145 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:27:51.148088 6145 catalog_manager.cc:1347] Generated new cluster ID: dc8e43b97c0e4554b1a7120a2ebc6ff0
I20260929 20:27:51.148216 6145 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:27:51.158723 6145 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 20:27:51.160681 6145 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:27:51.169631 6145 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6fd39200796944bab2bd9c348d191aa9: Generated new TSK 0
I20260929 20:27:51.170678 6145 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 20:27:51.184525 17621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu pid 6060
W20260929 20:27:51.185701 6122 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:58499
W20260929 20:27:51.186502 17621 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu pid 6060: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 20:27:51.186674 17621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu with pid 6060
2026-09-29T20:27:51Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1931 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790713339161492-17621-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 20:27:51 dist-test-slave-zbnc krb5kdc[6158](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 20:27:51 dist-test-slave-zbnc krb5kdc[6158](info): set up 2 sockets
Sep 29 20:27:51 dist-test-slave-zbnc krb5kdc[6158](info): commencing operation
krb5kdc: starting...
W20260929 20:27:52.296774 17621 mini_kdc.cc:121] Time spent starting KDC: real 1.072s user 0.002s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790713339161492-17621-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790713339161492-17621-0/krb5kdc/test-user.keytab.
Sep 29 20:27:52 dist-test-slave-zbnc krb5kdc[6158](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713672, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T20:27:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T20:27:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790713339161492-17621-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790713339161492-17621-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 20:27:52.560194 17621 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
/tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790713339161492-17621-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790713339161492-17621-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790713339161492-17621-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790713339161492-17621-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46817
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790713339161492-17621-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46817
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790713339161492-17621-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790713339161492-17621-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790713339161492-17621-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:27:52.915482 6171 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:27:52.915777 6171 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:27:52.915870 6171 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:27:52.923051 6171 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:27:52.923177 6171 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:27:52.923224 6171 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:27:52.923336 6171 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:27:52.923396 6171 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:27:52.933439 6171 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790713339161492-17621-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790713339161492-17621-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46817
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790713339161492-17621-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46817
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790713339161492-17621-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790713339161492-17621-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:27:52.934827 6171 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:27:52.936153 6171 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:27:52.948525 6176 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:27:52.948576 6177 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:27:52.948601 6179 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:27:52.949721 6171 server_base.cc:1034] running on GCE node
I20260929 20:27:52.950481 6171 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:27:52.951937 6171 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:27:52.953164 6171 hybrid_clock.cc:648] HybridClock initialized: now 1790713672953110 us; error 52 us; skew 500 ppm
I20260929 20:27:52.953521 6171 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:27:52 dist-test-slave-zbnc krb5kdc[6158](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713672, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:27:52.956759 6171 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 20:27:52.958148 6171 webserver.cc:460] Webserver started at http://127.0.0.1:38973/ using document root <none> and password file <none>
I20260929 20:27:52.958774 6171 fs_manager.cc:359] Metadata directory not provided
I20260929 20:27:52.958925 6171 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:27:52.959230 6171 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:27:52.962302 6171 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790713339161492-17621-0/minicluster-data/master-0/data/instance:
uuid: "b2eed1b2372842a980e2cde951f871a3"
format_stamp: "Formatted at 2026-09-29 20:27:52 on dist-test-slave-zbnc"
server_key: "7a0095e09a6df033fd7172fcee58c981"
server_key_iv: "18a8f234587b8e3f7d290ab6770951e2"
server_key_version: "encryptionkey@0"
I20260929 20:27:52.963063 6171 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790713339161492-17621-0/minicluster-data/master-0/wal/instance:
uuid: "b2eed1b2372842a980e2cde951f871a3"
format_stamp: "Formatted at 2026-09-29 20:27:52 on dist-test-slave-zbnc"
server_key: "7a0095e09a6df033fd7172fcee58c981"
server_key_iv: "18a8f234587b8e3f7d290ab6770951e2"
server_key_version: "encryptionkey@0"
I20260929 20:27:52.968034 6171 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260929 20:27:52.971426 6186 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:27:52.972939 6171 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260929 20:27:52.973078 6171 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790713339161492-17621-0/minicluster-data/master-0/data,/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790713339161492-17621-0/minicluster-data/master-0/wal
uuid: "b2eed1b2372842a980e2cde951f871a3"
format_stamp: "Formatted at 2026-09-29 20:27:52 on dist-test-slave-zbnc"
server_key: "7a0095e09a6df033fd7172fcee58c981"
server_key_iv: "18a8f234587b8e3f7d290ab6770951e2"
server_key_version: "encryptionkey@0"
I20260929 20:27:52.973255 6171 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790713339161492-17621-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790713339161492-17621-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790713339161492-17621-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:27:52.978998 6171 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:27:52.982931 6171 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:27:52.983177 6171 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:27:53.001154 6171 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46817
I20260929 20:27:53.001242 6237 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46817 every 8 connection(s)
I20260929 20:27:53.002568 6171 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790713339161492-17621-0/minicluster-data/master-0/data/info.pb
I20260929 20:27:53.005455 17621 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu as pid 6171
I20260929 20:27:53.005645 17621 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790713339161492-17621-0/minicluster-data/master-0/wal/instance
I20260929 20:27:53.006202 17621 external_mini_cluster.cc:1428] Setting key 502abfcab047da19d75b58d6c472e3ab
I20260929 20:27:53.007026 6238 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:27:53.014078 6241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:27:53.008165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33129 (local address 127.0.0.1:46817)
0929 20:27:53.008954 (+ 789us) server_negotiation.cc:206] Beginning negotiation
0929 20:27:53.008965 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 20:27:53.009003 (+ 38us) server_negotiation.cc:407] Connection header received
0929 20:27:53.010541 (+ 1538us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:27:53.010562 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:27:53.010996 (+ 434us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:27:53.011417 (+ 421us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:27:53.012485 (+ 1068us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:27:53.012491 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:27:53.012503 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:27:53.012516 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:27:53.012586 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:27:53.012612 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:27:53.012848 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0929 20:27:53.012991 (+ 143us) server_negotiation.cc:299] Negotiation successful
0929 20:27:53.013175 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":570,"thread_start_us":337,"threads_started":1}
I20260929 20:27:53.018332 17621 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 20:27:53.020864 6238 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b2eed1b2372842a980e2cde951f871a3: Bootstrap starting.
I20260929 20:27:53.024511 6238 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b2eed1b2372842a980e2cde951f871a3: Neither blocks nor log segments found. Creating new log.
I20260929 20:27:53.025718 6238 log.cc:824] T 00000000000000000000000000000000 P b2eed1b2372842a980e2cde951f871a3: Log is configured to *not* fsync() on all Append() calls
I20260929 20:27:53.025949 6241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:27:53.023142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57223 (local address 127.0.0.1:46817)
0929 20:27:53.023365 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0929 20:27:53.023369 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:27:53.023840 (+ 471us) server_negotiation.cc:407] Connection header received
0929 20:27:53.024057 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:27:53.024061 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:27:53.024110 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:27:53.024223 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:27:53.025231 (+ 1008us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:27:53.025237 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:27:53.025241 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:27:53.025259 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:27:53.025307 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:27:53.025324 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:27:53.025452 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0929 20:27:53.025551 (+ 99us) server_negotiation.cc:299] Negotiation successful
0929 20:27:53.025609 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260929 20:27:53.028931 6238 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b2eed1b2372842a980e2cde951f871a3: No bootstrap required, opened a new log
I20260929 20:27:53.036628 6238 raft_consensus.cc:348] T 00000000000000000000000000000000 P b2eed1b2372842a980e2cde951f871a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2eed1b2372842a980e2cde951f871a3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46817 } }
I20260929 20:27:53.037014 6238 raft_consensus.cc:374] T 00000000000000000000000000000000 P b2eed1b2372842a980e2cde951f871a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:27:53.037132 6238 raft_consensus.cc:729] T 00000000000000000000000000000000 P b2eed1b2372842a980e2cde951f871a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2eed1b2372842a980e2cde951f871a3, State: Initialized, Role: FOLLOWER
I20260929 20:27:53.037694 6238 consensus_queue.cc:260] T 00000000000000000000000000000000 P b2eed1b2372842a980e2cde951f871a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2eed1b2372842a980e2cde951f871a3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46817 } }
I20260929 20:27:53.037926 6238 raft_consensus.cc:388] T 00000000000000000000000000000000 P b2eed1b2372842a980e2cde951f871a3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 20:27:53.038060 6238 raft_consensus.cc:482] T 00000000000000000000000000000000 P b2eed1b2372842a980e2cde951f871a3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 20:27:53.038244 6238 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b2eed1b2372842a980e2cde951f871a3 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:27:53.040766 6238 raft_consensus.cc:504] T 00000000000000000000000000000000 P b2eed1b2372842a980e2cde951f871a3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2eed1b2372842a980e2cde951f871a3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46817 } }
I20260929 20:27:53.041410 6238 leader_election.cc:302] T 00000000000000000000000000000000 P b2eed1b2372842a980e2cde951f871a3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b2eed1b2372842a980e2cde951f871a3; no voters:
I20260929 20:27:53.042002 6238 leader_election.cc:288] T 00000000000000000000000000000000 P b2eed1b2372842a980e2cde951f871a3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 20:27:53.042158 6248 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b2eed1b2372842a980e2cde951f871a3 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:27:53.042884 6248 raft_consensus.cc:686] T 00000000000000000000000000000000 P b2eed1b2372842a980e2cde951f871a3 [term 1 LEADER]: Becoming Leader. State: Replica: b2eed1b2372842a980e2cde951f871a3, State: Running, Role: LEADER
I20260929 20:27:53.043447 6248 consensus_queue.cc:237] T 00000000000000000000000000000000 P b2eed1b2372842a980e2cde951f871a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2eed1b2372842a980e2cde951f871a3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46817 } }
I20260929 20:27:53.044104 6238 sys_catalog.cc:564] T 00000000000000000000000000000000 P b2eed1b2372842a980e2cde951f871a3 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:27:53.046504 6250 sys_catalog.cc:455] T 00000000000000000000000000000000 P b2eed1b2372842a980e2cde951f871a3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b2eed1b2372842a980e2cde951f871a3. Latest consensus state: current_term: 1 leader_uuid: "b2eed1b2372842a980e2cde951f871a3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2eed1b2372842a980e2cde951f871a3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46817 } } }
I20260929 20:27:53.046700 6249 sys_catalog.cc:455] T 00000000000000000000000000000000 P b2eed1b2372842a980e2cde951f871a3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b2eed1b2372842a980e2cde951f871a3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b2eed1b2372842a980e2cde951f871a3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46817 } } }
I20260929 20:27:53.047001 6250 sys_catalog.cc:458] T 00000000000000000000000000000000 P b2eed1b2372842a980e2cde951f871a3 [sys.catalog]: This master's current role is: LEADER
I20260929 20:27:53.047009 6249 sys_catalog.cc:458] T 00000000000000000000000000000000 P b2eed1b2372842a980e2cde951f871a3 [sys.catalog]: This master's current role is: LEADER
I20260929 20:27:53.049520 6257 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:27:53.055315 6257 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:27:53.065364 6257 catalog_manager.cc:1347] Generated new cluster ID: 11c34b974162443ca3c46e3cce2cb2af
I20260929 20:27:53.065485 6257 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:27:53.101046 6257 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 20:27:53.101616 6257 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:27:53.110419 6257 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b2eed1b2372842a980e2cde951f871a3: Generated new TSK 0
I20260929 20:27:53.111445 6257 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 20:27:53.126223 17621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu pid 6171
W20260929 20:27:53.127382 6233 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:33129
W20260929 20:27:53.128108 17621 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu pid 6171: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 20:27:53.128270 17621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu with pid 6171
2026-09-29T20:27:53Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1940 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790713339161492-17621-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 20:27:53 dist-test-slave-zbnc krb5kdc[6269](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 20:27:53 dist-test-slave-zbnc krb5kdc[6269](info): set up 2 sockets
Sep 29 20:27:53 dist-test-slave-zbnc krb5kdc[6269](info): commencing operation
krb5kdc: starting...
W20260929 20:27:54.245144 17621 mini_kdc.cc:121] Time spent starting KDC: real 1.080s user 0.000s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790713339161492-17621-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790713339161492-17621-0/krb5kdc/test-user.keytab.
Sep 29 20:27:54 dist-test-slave-zbnc krb5kdc[6269](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713674, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T20:27:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T20:27:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790713339161492-17621-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790713339161492-17621-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 20:27:54.514098 17621 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
/tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790713339161492-17621-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790713339161492-17621-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790713339161492-17621-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790713339161492-17621-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37385
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790713339161492-17621-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40497
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37385
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790713339161492-17621-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790713339161492-17621-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790713339161492-17621-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:27:54.850837 6282 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:27:54.851146 6282 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:27:54.851315 6282 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:27:54.858561 6282 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:27:54.858670 6282 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:27:54.858788 6282 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:27:54.858951 6282 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:27:54.859031 6282 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:27:54.869076 6282 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40497
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790713339161492-17621-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790713339161492-17621-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37385
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790713339161492-17621-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37385
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790713339161492-17621-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790713339161492-17621-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:27:54.870455 6282 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:27:54.871837 6282 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:27:54.884176 6288 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:27:54.884143 6290 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:27:54.884334 6287 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:27:54.884785 6282 server_base.cc:1034] running on GCE node
I20260929 20:27:54.885605 6282 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:27:54.887159 6282 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:27:54.888365 6282 hybrid_clock.cc:648] HybridClock initialized: now 1790713674888328 us; error 27 us; skew 500 ppm
I20260929 20:27:54.888751 6282 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:27:54 dist-test-slave-zbnc krb5kdc[6269](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713674, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:27:54.892079 6282 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 20:27:54.893640 6282 webserver.cc:460] Webserver started at http://127.0.0.1:43127/ using document root <none> and password file <none>
I20260929 20:27:54.894347 6282 fs_manager.cc:359] Metadata directory not provided
I20260929 20:27:54.894500 6282 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:27:54.894821 6282 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:27:54.897954 6282 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790713339161492-17621-0/minicluster-data/master-0/data/instance:
uuid: "32fe32d5dc2d4592a9ae5b1849f194dc"
format_stamp: "Formatted at 2026-09-29 20:27:54 on dist-test-slave-zbnc"
server_key: "0e1e8fb2f0618752d46b6d34de1327ab"
server_key_iv: "7b4dbcb5c2d8409b208c42a327ecdb08"
server_key_version: "encryptionkey@0"
I20260929 20:27:54.898752 6282 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790713339161492-17621-0/minicluster-data/master-0/wal/instance:
uuid: "32fe32d5dc2d4592a9ae5b1849f194dc"
format_stamp: "Formatted at 2026-09-29 20:27:54 on dist-test-slave-zbnc"
server_key: "0e1e8fb2f0618752d46b6d34de1327ab"
server_key_iv: "7b4dbcb5c2d8409b208c42a327ecdb08"
server_key_version: "encryptionkey@0"
I20260929 20:27:54.904440 6282 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260929 20:27:54.907959 6297 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:27:54.909258 6282 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 20:27:54.909446 6282 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790713339161492-17621-0/minicluster-data/master-0/data,/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790713339161492-17621-0/minicluster-data/master-0/wal
uuid: "32fe32d5dc2d4592a9ae5b1849f194dc"
format_stamp: "Formatted at 2026-09-29 20:27:54 on dist-test-slave-zbnc"
server_key: "0e1e8fb2f0618752d46b6d34de1327ab"
server_key_iv: "7b4dbcb5c2d8409b208c42a327ecdb08"
server_key_version: "encryptionkey@0"
I20260929 20:27:54.909646 6282 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790713339161492-17621-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790713339161492-17621-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790713339161492-17621-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:27:54.945318 6282 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:27:54.949810 6282 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:27:54.950109 6282 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:27:54.969198 6282 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37385
I20260929 20:27:54.969201 6348 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37385 every 8 connection(s)
I20260929 20:27:54.970667 6282 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790713339161492-17621-0/minicluster-data/master-0/data/info.pb
I20260929 20:27:54.974509 6349 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:27:54.980486 17621 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu as pid 6282
I20260929 20:27:54.980682 17621 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790713339161492-17621-0/minicluster-data/master-0/wal/instance
I20260929 20:27:54.981280 17621 external_mini_cluster.cc:1428] Setting key 2434a598da4bad78fe41471ef4390d81
I20260929 20:27:54.985846 6349 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 32fe32d5dc2d4592a9ae5b1849f194dc: Bootstrap starting.
I20260929 20:27:54.989693 6349 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 32fe32d5dc2d4592a9ae5b1849f194dc: Neither blocks nor log segments found. Creating new log.
I20260929 20:27:54.990976 6349 log.cc:824] T 00000000000000000000000000000000 P 32fe32d5dc2d4592a9ae5b1849f194dc: Log is configured to *not* fsync() on all Append() calls
I20260929 20:27:54.994235 6349 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 32fe32d5dc2d4592a9ae5b1849f194dc: No bootstrap required, opened a new log
Sep 29 20:27:54 dist-test-slave-zbnc krb5kdc[6269](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713674, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260929 20:27:55.002601 6349 raft_consensus.cc:348] T 00000000000000000000000000000000 P 32fe32d5dc2d4592a9ae5b1849f194dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32fe32d5dc2d4592a9ae5b1849f194dc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37385 } }
I20260929 20:27:55.003052 6349 raft_consensus.cc:374] T 00000000000000000000000000000000 P 32fe32d5dc2d4592a9ae5b1849f194dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:27:55.003170 6349 raft_consensus.cc:729] T 00000000000000000000000000000000 P 32fe32d5dc2d4592a9ae5b1849f194dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32fe32d5dc2d4592a9ae5b1849f194dc, State: Initialized, Role: FOLLOWER
I20260929 20:27:55.003835 6349 consensus_queue.cc:260] T 00000000000000000000000000000000 P 32fe32d5dc2d4592a9ae5b1849f194dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32fe32d5dc2d4592a9ae5b1849f194dc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37385 } }
I20260929 20:27:55.004096 6349 raft_consensus.cc:388] T 00000000000000000000000000000000 P 32fe32d5dc2d4592a9ae5b1849f194dc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 20:27:55.004256 6349 raft_consensus.cc:482] T 00000000000000000000000000000000 P 32fe32d5dc2d4592a9ae5b1849f194dc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 20:27:55.004464 6349 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 32fe32d5dc2d4592a9ae5b1849f194dc [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:27:55.005570 6352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:27:54.983377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.118:39597 (local address 127.0.0.1:37385)
0929 20:27:54.984208 (+ 831us) server_negotiation.cc:206] Beginning negotiation
0929 20:27:54.984220 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 20:27:54.984263 (+ 43us) server_negotiation.cc:407] Connection header received
0929 20:27:54.985945 (+ 1682us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:27:54.985977 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:27:54.986402 (+ 425us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:27:54.986722 (+ 320us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:27:54.988561 (+ 1839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:27:54.990035 (+ 1474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:27:54.991885 (+ 1850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:27:54.992730 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:27:54.992868 (+ 138us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:27:54.997775 (+ 4907us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:27:54.997782 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:27:54.997806 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:27:54.997852 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:27:55.001104 (+ 3252us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:27:55.001940 (+ 836us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:27:55.001946 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:27:55.001953 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:27:55.002117 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:27:55.002605 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:27:55.002609 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:27:55.002611 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:27:55.003321 (+ 710us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:27:55.003675 (+ 354us) server_negotiation.cc:1035] Waiting for connection context
0929 20:27:55.004155 (+ 480us) server_negotiation.cc:299] Negotiation successful
0929 20:27:55.004491 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":621,"thread_start_us":404,"threads_started":1}
I20260929 20:27:55.006983 6349 raft_consensus.cc:504] T 00000000000000000000000000000000 P 32fe32d5dc2d4592a9ae5b1849f194dc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32fe32d5dc2d4592a9ae5b1849f194dc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37385 } }
I20260929 20:27:55.007483 6349 leader_election.cc:302] T 00000000000000000000000000000000 P 32fe32d5dc2d4592a9ae5b1849f194dc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 32fe32d5dc2d4592a9ae5b1849f194dc; no voters:
I20260929 20:27:55.008145 6349 leader_election.cc:288] T 00000000000000000000000000000000 P 32fe32d5dc2d4592a9ae5b1849f194dc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 20:27:55.008289 6354 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 32fe32d5dc2d4592a9ae5b1849f194dc [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:27:55.009470 6354 raft_consensus.cc:686] T 00000000000000000000000000000000 P 32fe32d5dc2d4592a9ae5b1849f194dc [term 1 LEADER]: Becoming Leader. State: Replica: 32fe32d5dc2d4592a9ae5b1849f194dc, State: Running, Role: LEADER
I20260929 20:27:55.009570 17621 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 20:27:55.010038 6354 consensus_queue.cc:237] T 00000000000000000000000000000000 P 32fe32d5dc2d4592a9ae5b1849f194dc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32fe32d5dc2d4592a9ae5b1849f194dc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37385 } }
I20260929 20:27:55.010527 6349 sys_catalog.cc:564] T 00000000000000000000000000000000 P 32fe32d5dc2d4592a9ae5b1849f194dc [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:27:55.013044 6355 sys_catalog.cc:455] T 00000000000000000000000000000000 P 32fe32d5dc2d4592a9ae5b1849f194dc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "32fe32d5dc2d4592a9ae5b1849f194dc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32fe32d5dc2d4592a9ae5b1849f194dc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37385 } } }
I20260929 20:27:55.013526 6355 sys_catalog.cc:458] T 00000000000000000000000000000000 P 32fe32d5dc2d4592a9ae5b1849f194dc [sys.catalog]: This master's current role is: LEADER
I20260929 20:27:55.014308 6356 sys_catalog.cc:455] T 00000000000000000000000000000000 P 32fe32d5dc2d4592a9ae5b1849f194dc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 32fe32d5dc2d4592a9ae5b1849f194dc. Latest consensus state: current_term: 1 leader_uuid: "32fe32d5dc2d4592a9ae5b1849f194dc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32fe32d5dc2d4592a9ae5b1849f194dc" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37385 } } }
I20260929 20:27:55.014573 6356 sys_catalog.cc:458] T 00000000000000000000000000000000 P 32fe32d5dc2d4592a9ae5b1849f194dc [sys.catalog]: This master's current role is: LEADER
I20260929 20:27:55.015941 6360 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:27:55.022616 6360 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:27:55.044379 6360 catalog_manager.cc:1347] Generated new cluster ID: d0991687181f4d6a9cd56279dcabeeeb
I20260929 20:27:55.044507 6360 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:27:55.049352 6352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:27:55.018193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.118:59005 (local address 127.0.0.1:37385)
0929 20:27:55.018362 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0929 20:27:55.018371 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 20:27:55.025544 (+ 7173us) server_negotiation.cc:407] Connection header received
0929 20:27:55.026143 (+ 599us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:27:55.026157 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:27:55.026291 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:27:55.026395 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:27:55.029199 (+ 2804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:27:55.030916 (+ 1717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:27:55.033525 (+ 2609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:27:55.034447 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:27:55.034572 (+ 125us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:27:55.039116 (+ 4544us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:27:55.039130 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:27:55.039137 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:27:55.039198 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:27:55.044286 (+ 5088us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:27:55.045676 (+ 1390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:27:55.045685 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:27:55.045692 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:27:55.045883 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:27:55.046664 (+ 781us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:27:55.046672 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:27:55.046677 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:27:55.047294 (+ 617us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:27:55.047645 (+ 351us) server_negotiation.cc:1035] Waiting for connection context
0929 20:27:55.048540 (+ 895us) server_negotiation.cc:299] Negotiation successful
0929 20:27:55.048792 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260929 20:27:55.059101 6360 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 20:27:55.061056 6360 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:27:55.070568 6360 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 32fe32d5dc2d4592a9ae5b1849f194dc: Generated new TSK 0
I20260929 20:27:55.071695 6360 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 20:27:55.078028 17621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu pid 6282
I20260929 20:27:55.079775 6344 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:27:55.168797 17621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu with pid 6282
W20260929 20:27:55.186228 6267 connection.cc:535] client connection to 127.0.0.1:37385 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:37385): Connection reset by peer (error 104)
2026-09-29T20:27:55Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (2041 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790713339161492-17621-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 20:27:55 dist-test-slave-zbnc krb5kdc[6381](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 20:27:55 dist-test-slave-zbnc krb5kdc[6381](info): set up 2 sockets
Sep 29 20:27:55 dist-test-slave-zbnc krb5kdc[6381](info): commencing operation
krb5kdc: starting...
W20260929 20:27:56.280789 17621 mini_kdc.cc:121] Time spent starting KDC: real 1.074s user 0.005s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790713339161492-17621-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790713339161492-17621-0/krb5kdc/test-user.keytab.
Sep 29 20:27:56 dist-test-slave-zbnc krb5kdc[6381](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713676, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T20:27:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T20:27:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790713339161492-17621-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790713339161492-17621-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 20:27:56.542018 17621 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
/tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790713339161492-17621-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790713339161492-17621-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790713339161492-17621-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790713339161492-17621-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34069
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790713339161492-17621-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43523
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34069
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790713339161492-17621-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790713339161492-17621-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790713339161492-17621-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:27:56.886377 6394 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:27:56.886672 6394 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:27:56.886835 6394 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:27:56.894294 6394 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:27:56.894481 6394 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:27:56.894593 6394 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:27:56.894706 6394 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:27:56.894801 6394 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:27:56.905902 6394 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43523
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790713339161492-17621-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790713339161492-17621-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34069
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790713339161492-17621-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34069
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790713339161492-17621-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790713339161492-17621-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:27:56.907395 6394 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:27:56.908754 6394 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:27:56.920918 6399 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:27:56.920884 6400 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:27:56.920944 6402 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:27:56.922132 6394 server_base.cc:1034] running on GCE node
I20260929 20:27:56.922906 6394 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:27:56.924304 6394 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:27:56.925544 6394 hybrid_clock.cc:648] HybridClock initialized: now 1790713676925512 us; error 58 us; skew 500 ppm
I20260929 20:27:56.925917 6394 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:27:56 dist-test-slave-zbnc krb5kdc[6381](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713676, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:27:56.929059 6394 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 20:27:56.930487 6394 webserver.cc:460] Webserver started at http://127.0.0.1:42791/ using document root <none> and password file <none>
I20260929 20:27:56.931227 6394 fs_manager.cc:359] Metadata directory not provided
I20260929 20:27:56.931375 6394 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:27:56.931680 6394 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:27:56.934648 6394 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790713339161492-17621-0/minicluster-data/master-0/data/instance:
uuid: "91a2a337a7ea49cbb1cffe0188ebbf01"
format_stamp: "Formatted at 2026-09-29 20:27:56 on dist-test-slave-zbnc"
server_key: "5bdc249171c096e170be1a938b4845fb"
server_key_iv: "2c0272f7bf0a8139de0cb605a7107631"
server_key_version: "encryptionkey@0"
I20260929 20:27:56.935484 6394 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790713339161492-17621-0/minicluster-data/master-0/wal/instance:
uuid: "91a2a337a7ea49cbb1cffe0188ebbf01"
format_stamp: "Formatted at 2026-09-29 20:27:56 on dist-test-slave-zbnc"
server_key: "5bdc249171c096e170be1a938b4845fb"
server_key_iv: "2c0272f7bf0a8139de0cb605a7107631"
server_key_version: "encryptionkey@0"
I20260929 20:27:56.940521 6394 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260929 20:27:56.944028 6409 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:27:56.945293 6394 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 20:27:56.945487 6394 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790713339161492-17621-0/minicluster-data/master-0/data,/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790713339161492-17621-0/minicluster-data/master-0/wal
uuid: "91a2a337a7ea49cbb1cffe0188ebbf01"
format_stamp: "Formatted at 2026-09-29 20:27:56 on dist-test-slave-zbnc"
server_key: "5bdc249171c096e170be1a938b4845fb"
server_key_iv: "2c0272f7bf0a8139de0cb605a7107631"
server_key_version: "encryptionkey@0"
I20260929 20:27:56.945688 6394 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790713339161492-17621-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790713339161492-17621-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790713339161492-17621-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:27:56.974133 6394 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:27:56.978742 6394 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:27:56.979077 6394 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:27:56.997906 6394 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34069
I20260929 20:27:56.997996 6460 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34069 every 8 connection(s)
I20260929 20:27:56.999353 6394 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790713339161492-17621-0/minicluster-data/master-0/data/info.pb
I20260929 20:27:57.003181 6461 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:27:57.007694 17621 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu as pid 6394
I20260929 20:27:57.007906 17621 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790713339161492-17621-0/minicluster-data/master-0/wal/instance
I20260929 20:27:57.008457 17621 external_mini_cluster.cc:1428] Setting key 71f60ebb5beabccb5a9430b9a1626fd1
I20260929 20:27:57.016654 6461 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 91a2a337a7ea49cbb1cffe0188ebbf01: Bootstrap starting.
I20260929 20:27:57.020428 6461 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 91a2a337a7ea49cbb1cffe0188ebbf01: Neither blocks nor log segments found. Creating new log.
I20260929 20:27:57.021536 6461 log.cc:824] T 00000000000000000000000000000000 P 91a2a337a7ea49cbb1cffe0188ebbf01: Log is configured to *not* fsync() on all Append() calls
I20260929 20:27:57.021689 6464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:27:57.010257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.118:57467 (local address 127.0.0.1:34069)
0929 20:27:57.011040 (+ 783us) server_negotiation.cc:206] Beginning negotiation
0929 20:27:57.011052 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 20:27:57.011096 (+ 44us) server_negotiation.cc:407] Connection header received
0929 20:27:57.012628 (+ 1532us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:27:57.012650 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:27:57.013075 (+ 425us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:27:57.013391 (+ 316us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:27:57.013995 (+ 604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:27:57.015379 (+ 1384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:27:57.017209 (+ 1830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:27:57.017852 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:27:57.017998 (+ 146us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:27:57.019424 (+ 1426us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:27:57.019432 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:27:57.019445 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:27:57.019955 (+ 510us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:27:57.020025 (+ 70us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:27:57.020055 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:27:57.020168 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0929 20:27:57.020486 (+ 318us) server_negotiation.cc:299] Negotiation successful
0929 20:27:57.020696 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":542,"thread_start_us":348,"threads_started":1}
I20260929 20:27:57.024603 6461 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 91a2a337a7ea49cbb1cffe0188ebbf01: No bootstrap required, opened a new log
I20260929 20:27:57.025580 17621 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 20:27:57.032769 6461 raft_consensus.cc:348] T 00000000000000000000000000000000 P 91a2a337a7ea49cbb1cffe0188ebbf01 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91a2a337a7ea49cbb1cffe0188ebbf01" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34069 } }
I20260929 20:27:57.033165 6461 raft_consensus.cc:374] T 00000000000000000000000000000000 P 91a2a337a7ea49cbb1cffe0188ebbf01 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:27:57.033277 6461 raft_consensus.cc:729] T 00000000000000000000000000000000 P 91a2a337a7ea49cbb1cffe0188ebbf01 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91a2a337a7ea49cbb1cffe0188ebbf01, State: Initialized, Role: FOLLOWER
I20260929 20:27:57.033921 6461 consensus_queue.cc:260] T 00000000000000000000000000000000 P 91a2a337a7ea49cbb1cffe0188ebbf01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91a2a337a7ea49cbb1cffe0188ebbf01" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34069 } }
I20260929 20:27:57.034188 6461 raft_consensus.cc:388] T 00000000000000000000000000000000 P 91a2a337a7ea49cbb1cffe0188ebbf01 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 20:27:57.034344 6461 raft_consensus.cc:482] T 00000000000000000000000000000000 P 91a2a337a7ea49cbb1cffe0188ebbf01 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 20:27:57.034550 6461 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 91a2a337a7ea49cbb1cffe0188ebbf01 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:27:57.036293 6464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:27:57.029781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.118:47547 (local address 127.0.0.1:34069)
0929 20:27:57.029936 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0929 20:27:57.029941 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:27:57.030452 (+ 511us) server_negotiation.cc:407] Connection header received
0929 20:27:57.030665 (+ 213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:27:57.030669 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:27:57.030727 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:27:57.030837 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:27:57.031385 (+ 548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:27:57.032161 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:27:57.033683 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:27:57.034182 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:27:57.034328 (+ 146us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:27:57.035560 (+ 1232us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:27:57.035565 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:27:57.035568 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:27:57.035585 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:27:57.035624 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:27:57.035639 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:27:57.035770 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0929 20:27:57.035968 (+ 198us) server_negotiation.cc:299] Negotiation successful
0929 20:27:57.036024 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260929 20:27:57.037194 6461 raft_consensus.cc:504] T 00000000000000000000000000000000 P 91a2a337a7ea49cbb1cffe0188ebbf01 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91a2a337a7ea49cbb1cffe0188ebbf01" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34069 } }
I20260929 20:27:57.037794 6461 leader_election.cc:302] T 00000000000000000000000000000000 P 91a2a337a7ea49cbb1cffe0188ebbf01 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 91a2a337a7ea49cbb1cffe0188ebbf01; no voters:
I20260929 20:27:57.038388 6461 leader_election.cc:288] T 00000000000000000000000000000000 P 91a2a337a7ea49cbb1cffe0188ebbf01 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 20:27:57.038535 6471 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 91a2a337a7ea49cbb1cffe0188ebbf01 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:27:57.039326 6471 raft_consensus.cc:686] T 00000000000000000000000000000000 P 91a2a337a7ea49cbb1cffe0188ebbf01 [term 1 LEADER]: Becoming Leader. State: Replica: 91a2a337a7ea49cbb1cffe0188ebbf01, State: Running, Role: LEADER
I20260929 20:27:57.039908 6471 consensus_queue.cc:237] T 00000000000000000000000000000000 P 91a2a337a7ea49cbb1cffe0188ebbf01 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91a2a337a7ea49cbb1cffe0188ebbf01" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34069 } }
I20260929 20:27:57.040678 6461 sys_catalog.cc:564] T 00000000000000000000000000000000 P 91a2a337a7ea49cbb1cffe0188ebbf01 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:27:57.043174 6472 sys_catalog.cc:455] T 00000000000000000000000000000000 P 91a2a337a7ea49cbb1cffe0188ebbf01 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "91a2a337a7ea49cbb1cffe0188ebbf01" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91a2a337a7ea49cbb1cffe0188ebbf01" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34069 } } }
I20260929 20:27:57.043138 6473 sys_catalog.cc:455] T 00000000000000000000000000000000 P 91a2a337a7ea49cbb1cffe0188ebbf01 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 91a2a337a7ea49cbb1cffe0188ebbf01. Latest consensus state: current_term: 1 leader_uuid: "91a2a337a7ea49cbb1cffe0188ebbf01" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "91a2a337a7ea49cbb1cffe0188ebbf01" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34069 } } }
I20260929 20:27:57.043596 6473 sys_catalog.cc:458] T 00000000000000000000000000000000 P 91a2a337a7ea49cbb1cffe0188ebbf01 [sys.catalog]: This master's current role is: LEADER
I20260929 20:27:57.043596 6472 sys_catalog.cc:458] T 00000000000000000000000000000000 P 91a2a337a7ea49cbb1cffe0188ebbf01 [sys.catalog]: This master's current role is: LEADER
I20260929 20:27:57.044552 6480 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:27:57.050741 6480 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:27:57.065788 6480 catalog_manager.cc:1347] Generated new cluster ID: 67351ec14a83496f91f60b2dd32dae25
I20260929 20:27:57.066136 6480 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:27:57.075728 6480 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 20:27:57.077593 6480 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:27:57.100132 6480 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 91a2a337a7ea49cbb1cffe0188ebbf01: Generated new TSK 0
I20260929 20:27:57.101303 6480 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 20:27:57.109292 17621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu pid 6394
W20260929 20:27:57.110574 6456 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.118:57467
W20260929 20:27:57.111442 17621 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu pid 6394: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 20:27:57.111617 17621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu with pid 6394
W20260929 20:27:57.128952 6379 connection.cc:535] client connection to 127.0.0.1:34069 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:34069): Connection reset by peer (error 104)
2026-09-29T20:27:57Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1942 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790713339161492-17621-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 20:27:57 dist-test-slave-zbnc krb5kdc[6492](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 20:27:57 dist-test-slave-zbnc krb5kdc[6492](info): set up 2 sockets
Sep 29 20:27:57 dist-test-slave-zbnc krb5kdc[6492](info): commencing operation
krb5kdc: starting...
W20260929 20:27:58.224763 17621 mini_kdc.cc:121] Time spent starting KDC: real 1.076s user 0.000s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790713339161492-17621-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790713339161492-17621-0/krb5kdc/test-user.keytab.
Sep 29 20:27:58 dist-test-slave-zbnc krb5kdc[6492](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713678, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T20:27:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T20:27:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790713339161492-17621-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790713339161492-17621-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 20:27:58.481936 17621 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
/tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790713339161492-17621-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790713339161492-17621-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790713339161492-17621-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790713339161492-17621-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35931
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790713339161492-17621-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44241
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35931
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790713339161492-17621-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790713339161492-17621-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790713339161492-17621-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:27:58.815462 6505 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:27:58.815724 6505 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:27:58.815876 6505 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:27:58.823132 6505 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:27:58.823267 6505 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:27:58.823375 6505 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:27:58.823479 6505 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:27:58.823556 6505 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:27:58.833475 6505 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44241
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790713339161492-17621-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790713339161492-17621-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35931
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790713339161492-17621-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35931
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790713339161492-17621-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790713339161492-17621-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:27:58.834789 6505 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:27:58.836167 6505 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:27:58.848412 6513 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:27:58.848450 6511 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:27:58.848425 6510 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:27:58.849226 6505 server_base.cc:1034] running on GCE node
I20260929 20:27:58.850016 6505 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:27:58.851613 6505 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:27:58.852850 6505 hybrid_clock.cc:648] HybridClock initialized: now 1790713678852813 us; error 50 us; skew 500 ppm
I20260929 20:27:58.853210 6505 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:27:58 dist-test-slave-zbnc krb5kdc[6492](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713678, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:27:58.861511 6505 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 20:27:58.863214 6505 webserver.cc:460] Webserver started at http://127.0.0.1:34233/ using document root <none> and password file <none>
I20260929 20:27:58.863946 6505 fs_manager.cc:359] Metadata directory not provided
I20260929 20:27:58.864095 6505 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:27:58.864424 6505 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:27:58.867750 6505 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790713339161492-17621-0/minicluster-data/master-0/data/instance:
uuid: "fc4dabbb0fb7483c9ba9003cb55f2f8d"
format_stamp: "Formatted at 2026-09-29 20:27:58 on dist-test-slave-zbnc"
server_key: "8b2a1085634f7b0d28816a9df0af1339"
server_key_iv: "d5223d2d4293f45333ad4c9dfc0ae3b9"
server_key_version: "encryptionkey@0"
I20260929 20:27:58.868556 6505 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790713339161492-17621-0/minicluster-data/master-0/wal/instance:
uuid: "fc4dabbb0fb7483c9ba9003cb55f2f8d"
format_stamp: "Formatted at 2026-09-29 20:27:58 on dist-test-slave-zbnc"
server_key: "8b2a1085634f7b0d28816a9df0af1339"
server_key_iv: "d5223d2d4293f45333ad4c9dfc0ae3b9"
server_key_version: "encryptionkey@0"
I20260929 20:27:58.873978 6505 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260929 20:27:58.877704 6520 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:27:58.879086 6505 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 20:27:58.879269 6505 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790713339161492-17621-0/minicluster-data/master-0/data,/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790713339161492-17621-0/minicluster-data/master-0/wal
uuid: "fc4dabbb0fb7483c9ba9003cb55f2f8d"
format_stamp: "Formatted at 2026-09-29 20:27:58 on dist-test-slave-zbnc"
server_key: "8b2a1085634f7b0d28816a9df0af1339"
server_key_iv: "d5223d2d4293f45333ad4c9dfc0ae3b9"
server_key_version: "encryptionkey@0"
I20260929 20:27:58.879490 6505 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790713339161492-17621-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790713339161492-17621-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790713339161492-17621-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:27:58.885381 6505 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:27:58.889468 6505 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:27:58.889763 6505 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:27:58.908578 6505 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35931
I20260929 20:27:58.908617 6571 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35931 every 8 connection(s)
I20260929 20:27:58.910030 6505 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790713339161492-17621-0/minicluster-data/master-0/data/info.pb
I20260929 20:27:58.913806 6572 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:27:58.916713 17621 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu as pid 6505
I20260929 20:27:58.916896 17621 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790713339161492-17621-0/minicluster-data/master-0/wal/instance
I20260929 20:27:58.917346 17621 external_mini_cluster.cc:1428] Setting key a1003aaf4965512702ab40b7da853913
I20260929 20:27:58.925161 6575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:27:58.919290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.118:39285 (local address 127.0.0.1:35931)
0929 20:27:58.920106 (+ 816us) server_negotiation.cc:206] Beginning negotiation
0929 20:27:58.920119 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 20:27:58.920159 (+ 40us) server_negotiation.cc:407] Connection header received
0929 20:27:58.921782 (+ 1623us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:27:58.921804 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:27:58.922145 (+ 341us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:27:58.922540 (+ 395us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:27:58.923645 (+ 1105us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:27:58.923655 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:27:58.923667 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:27:58.923681 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:27:58.923736 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:27:58.923757 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:27:58.923931 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0929 20:27:58.924038 (+ 107us) server_negotiation.cc:299] Negotiation successful
0929 20:27:58.924249 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":606,"thread_start_us":380,"threads_started":1}
I20260929 20:27:58.926270 6572 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc4dabbb0fb7483c9ba9003cb55f2f8d: Bootstrap starting.
I20260929 20:27:58.929004 17621 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 20:27:58.929687 6572 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fc4dabbb0fb7483c9ba9003cb55f2f8d: Neither blocks nor log segments found. Creating new log.
I20260929 20:27:58.930663 6572 log.cc:824] T 00000000000000000000000000000000 P fc4dabbb0fb7483c9ba9003cb55f2f8d: Log is configured to *not* fsync() on all Append() calls
I20260929 20:27:58.933619 6572 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc4dabbb0fb7483c9ba9003cb55f2f8d: No bootstrap required, opened a new log
I20260929 20:27:58.936368 6575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:27:58.933816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.118:33467 (local address 127.0.0.1:35931)
0929 20:27:58.934039 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0929 20:27:58.934046 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:27:58.934418 (+ 372us) server_negotiation.cc:407] Connection header received
0929 20:27:58.934599 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:27:58.934603 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:27:58.934646 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:27:58.934717 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:27:58.935708 (+ 991us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:27:58.935712 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:27:58.935719 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:27:58.935733 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:27:58.935774 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:27:58.935789 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:27:58.935883 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0929 20:27:58.936003 (+ 120us) server_negotiation.cc:299] Negotiation successful
0929 20:27:58.936053 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 20:27:58.941689 6572 raft_consensus.cc:348] T 00000000000000000000000000000000 P fc4dabbb0fb7483c9ba9003cb55f2f8d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc4dabbb0fb7483c9ba9003cb55f2f8d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35931 } }
I20260929 20:27:58.942049 6572 raft_consensus.cc:374] T 00000000000000000000000000000000 P fc4dabbb0fb7483c9ba9003cb55f2f8d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:27:58.942160 6572 raft_consensus.cc:729] T 00000000000000000000000000000000 P fc4dabbb0fb7483c9ba9003cb55f2f8d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc4dabbb0fb7483c9ba9003cb55f2f8d, State: Initialized, Role: FOLLOWER
I20260929 20:27:58.942708 6572 consensus_queue.cc:260] T 00000000000000000000000000000000 P fc4dabbb0fb7483c9ba9003cb55f2f8d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc4dabbb0fb7483c9ba9003cb55f2f8d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35931 } }
I20260929 20:27:58.942961 6572 raft_consensus.cc:388] T 00000000000000000000000000000000 P fc4dabbb0fb7483c9ba9003cb55f2f8d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 20:27:58.943068 6572 raft_consensus.cc:482] T 00000000000000000000000000000000 P fc4dabbb0fb7483c9ba9003cb55f2f8d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 20:27:58.943193 6572 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fc4dabbb0fb7483c9ba9003cb55f2f8d [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:27:58.945657 6572 raft_consensus.cc:504] T 00000000000000000000000000000000 P fc4dabbb0fb7483c9ba9003cb55f2f8d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc4dabbb0fb7483c9ba9003cb55f2f8d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35931 } }
I20260929 20:27:58.946091 6572 leader_election.cc:302] T 00000000000000000000000000000000 P fc4dabbb0fb7483c9ba9003cb55f2f8d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fc4dabbb0fb7483c9ba9003cb55f2f8d; no voters:
I20260929 20:27:58.946615 6572 leader_election.cc:288] T 00000000000000000000000000000000 P fc4dabbb0fb7483c9ba9003cb55f2f8d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 20:27:58.946725 6582 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fc4dabbb0fb7483c9ba9003cb55f2f8d [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:27:58.947461 6582 raft_consensus.cc:686] T 00000000000000000000000000000000 P fc4dabbb0fb7483c9ba9003cb55f2f8d [term 1 LEADER]: Becoming Leader. State: Replica: fc4dabbb0fb7483c9ba9003cb55f2f8d, State: Running, Role: LEADER
I20260929 20:27:58.948004 6582 consensus_queue.cc:237] T 00000000000000000000000000000000 P fc4dabbb0fb7483c9ba9003cb55f2f8d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc4dabbb0fb7483c9ba9003cb55f2f8d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35931 } }
I20260929 20:27:58.948747 6572 sys_catalog.cc:564] T 00000000000000000000000000000000 P fc4dabbb0fb7483c9ba9003cb55f2f8d [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:27:58.950603 6584 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc4dabbb0fb7483c9ba9003cb55f2f8d [sys.catalog]: SysCatalogTable state changed. Reason: New leader fc4dabbb0fb7483c9ba9003cb55f2f8d. Latest consensus state: current_term: 1 leader_uuid: "fc4dabbb0fb7483c9ba9003cb55f2f8d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc4dabbb0fb7483c9ba9003cb55f2f8d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35931 } } }
I20260929 20:27:58.950944 6584 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc4dabbb0fb7483c9ba9003cb55f2f8d [sys.catalog]: This master's current role is: LEADER
I20260929 20:27:58.952069 6583 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc4dabbb0fb7483c9ba9003cb55f2f8d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fc4dabbb0fb7483c9ba9003cb55f2f8d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc4dabbb0fb7483c9ba9003cb55f2f8d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35931 } } }
I20260929 20:27:58.952857 6583 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc4dabbb0fb7483c9ba9003cb55f2f8d [sys.catalog]: This master's current role is: LEADER
I20260929 20:27:58.953013 6588 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:27:58.958236 6588 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:27:58.967733 6588 catalog_manager.cc:1347] Generated new cluster ID: 87d1081d26b94e479c734cb94152fdb1
I20260929 20:27:58.967869 6588 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:27:58.987504 6588 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 20:27:58.987941 6588 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:27:58.998418 6588 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fc4dabbb0fb7483c9ba9003cb55f2f8d: Generated new TSK 0
I20260929 20:27:58.999289 6588 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 20:27:59.008775 17621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu pid 6505
W20260929 20:27:59.009886 6567 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.118:39285
W20260929 20:27:59.010608 17621 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu pid 6505: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 20:27:59.010767 17621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu with pid 6505
2026-09-29T20:27:59Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1898 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790713339161492-17621-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 20:27:59 dist-test-slave-zbnc krb5kdc[6603](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 20:27:59 dist-test-slave-zbnc krb5kdc[6603](info): set up 2 sockets
Sep 29 20:27:59 dist-test-slave-zbnc krb5kdc[6603](info): commencing operation
krb5kdc: starting...
W20260929 20:28:00.120685 17621 mini_kdc.cc:121] Time spent starting KDC: real 1.073s user 0.002s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790713339161492-17621-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790713339161492-17621-0/krb5kdc/test-user.keytab.
Sep 29 20:28:00 dist-test-slave-zbnc krb5kdc[6603](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713680, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T20:28:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T20:28:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790713339161492-17621-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790713339161492-17621-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 20:28:00.380395 17621 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
/tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790713339161492-17621-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790713339161492-17621-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790713339161492-17621-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790713339161492-17621-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40481
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790713339161492-17621-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39267
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40481
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790713339161492-17621-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790713339161492-17621-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790713339161492-17621-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:28:00.716799 6616 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:28:00.717090 6616 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:28:00.717258 6616 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:28:00.724570 6616 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:28:00.724694 6616 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:28:00.724743 6616 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:28:00.724877 6616 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:28:00.724964 6616 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:28:00.735200 6616 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39267
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790713339161492-17621-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790713339161492-17621-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40481
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790713339161492-17621-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40481
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790713339161492-17621-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790713339161492-17621-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:28:00.736542 6616 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:28:00.737913 6616 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:28:00.750313 6624 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:28:00.750402 6622 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:28:00.750598 6621 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:28:00.751967 6616 server_base.cc:1034] running on GCE node
I20260929 20:28:00.752710 6616 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:28:00.754050 6616 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:28:00.755275 6616 hybrid_clock.cc:648] HybridClock initialized: now 1790713680755241 us; error 49 us; skew 500 ppm
I20260929 20:28:00.755635 6616 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:28:00 dist-test-slave-zbnc krb5kdc[6603](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713680, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:28:00.758623 6616 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 20:28:00.760126 6616 webserver.cc:460] Webserver started at http://127.0.0.1:41503/ using document root <none> and password file <none>
I20260929 20:28:00.760741 6616 fs_manager.cc:359] Metadata directory not provided
I20260929 20:28:00.760838 6616 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:28:00.761097 6616 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:28:00.764021 6616 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790713339161492-17621-0/minicluster-data/master-0/data/instance:
uuid: "e579829cd232471fbfef788138bcecad"
format_stamp: "Formatted at 2026-09-29 20:28:00 on dist-test-slave-zbnc"
server_key: "94901c246c0c57a0b79fb94afd139360"
server_key_iv: "7529020d5a0eabd9cb01cec6fa4347db"
server_key_version: "encryptionkey@0"
I20260929 20:28:00.764761 6616 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790713339161492-17621-0/minicluster-data/master-0/wal/instance:
uuid: "e579829cd232471fbfef788138bcecad"
format_stamp: "Formatted at 2026-09-29 20:28:00 on dist-test-slave-zbnc"
server_key: "94901c246c0c57a0b79fb94afd139360"
server_key_iv: "7529020d5a0eabd9cb01cec6fa4347db"
server_key_version: "encryptionkey@0"
I20260929 20:28:00.769789 6616 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260929 20:28:00.773276 6631 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:28:00.774676 6616 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 20:28:00.774825 6616 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790713339161492-17621-0/minicluster-data/master-0/data,/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790713339161492-17621-0/minicluster-data/master-0/wal
uuid: "e579829cd232471fbfef788138bcecad"
format_stamp: "Formatted at 2026-09-29 20:28:00 on dist-test-slave-zbnc"
server_key: "94901c246c0c57a0b79fb94afd139360"
server_key_iv: "7529020d5a0eabd9cb01cec6fa4347db"
server_key_version: "encryptionkey@0"
I20260929 20:28:00.775071 6616 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790713339161492-17621-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790713339161492-17621-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790713339161492-17621-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:28:00.801983 6616 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:28:00.806373 6616 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:28:00.806622 6616 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:28:00.825184 6616 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40481
I20260929 20:28:00.825206 6682 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40481 every 8 connection(s)
I20260929 20:28:00.826695 6616 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790713339161492-17621-0/minicluster-data/master-0/data/info.pb
I20260929 20:28:00.831307 6683 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:28:00.836205 17621 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu as pid 6616
I20260929 20:28:00.836395 17621 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790713339161492-17621-0/minicluster-data/master-0/wal/instance
I20260929 20:28:00.836881 17621 external_mini_cluster.cc:1428] Setting key beba360e46267d8a9db59360d739b94a
I20260929 20:28:00.847247 6683 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e579829cd232471fbfef788138bcecad: Bootstrap starting.
I20260929 20:28:00.851634 6683 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e579829cd232471fbfef788138bcecad: Neither blocks nor log segments found. Creating new log.
I20260929 20:28:00.852859 6683 log.cc:824] T 00000000000000000000000000000000 P e579829cd232471fbfef788138bcecad: Log is configured to *not* fsync() on all Append() calls
Sep 29 20:28:00 dist-test-slave-zbnc krb5kdc[6603](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713680, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260929 20:28:00.856699 6683 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e579829cd232471fbfef788138bcecad: No bootstrap required, opened a new log
I20260929 20:28:00.864077 6686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:28:00.838850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.118:33325 (local address 127.0.0.1:40481)
0929 20:28:00.839786 (+ 936us) server_negotiation.cc:206] Beginning negotiation
0929 20:28:00.839805 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0929 20:28:00.839871 (+ 66us) server_negotiation.cc:407] Connection header received
0929 20:28:00.842036 (+ 2165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:28:00.842076 (+ 40us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:28:00.842769 (+ 693us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:28:00.843350 (+ 581us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:28:00.844793 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:28:00.847092 (+ 2299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:28:00.848478 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:28:00.849758 (+ 1280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:28:00.850003 (+ 245us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:28:00.854676 (+ 4673us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:28:00.854687 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:28:00.854709 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:28:00.854760 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:28:00.858718 (+ 3958us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:28:00.859829 (+ 1111us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:28:00.859837 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:28:00.859845 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:28:00.860062 (+ 217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:28:00.860699 (+ 637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:28:00.860705 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:28:00.860708 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:28:00.861373 (+ 665us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:28:00.861786 (+ 413us) server_negotiation.cc:1035] Waiting for connection context
0929 20:28:00.862427 (+ 641us) server_negotiation.cc:299] Negotiation successful
0929 20:28:00.862821 (+ 394us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":619,"thread_start_us":344,"threads_started":1}
I20260929 20:28:00.864880 6683 raft_consensus.cc:348] T 00000000000000000000000000000000 P e579829cd232471fbfef788138bcecad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e579829cd232471fbfef788138bcecad" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40481 } }
I20260929 20:28:00.865326 6683 raft_consensus.cc:374] T 00000000000000000000000000000000 P e579829cd232471fbfef788138bcecad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:28:00.865437 6683 raft_consensus.cc:729] T 00000000000000000000000000000000 P e579829cd232471fbfef788138bcecad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e579829cd232471fbfef788138bcecad, State: Initialized, Role: FOLLOWER
I20260929 20:28:00.866022 6683 consensus_queue.cc:260] T 00000000000000000000000000000000 P e579829cd232471fbfef788138bcecad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e579829cd232471fbfef788138bcecad" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40481 } }
I20260929 20:28:00.866281 6683 raft_consensus.cc:388] T 00000000000000000000000000000000 P e579829cd232471fbfef788138bcecad [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 20:28:00.866431 6683 raft_consensus.cc:482] T 00000000000000000000000000000000 P e579829cd232471fbfef788138bcecad [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 20:28:00.866618 6683 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e579829cd232471fbfef788138bcecad [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:28:00.868887 17621 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 20:28:00.869416 6683 raft_consensus.cc:504] T 00000000000000000000000000000000 P e579829cd232471fbfef788138bcecad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e579829cd232471fbfef788138bcecad" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40481 } }
I20260929 20:28:00.869906 6683 leader_election.cc:302] T 00000000000000000000000000000000 P e579829cd232471fbfef788138bcecad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e579829cd232471fbfef788138bcecad; no voters:
I20260929 20:28:00.870605 6683 leader_election.cc:288] T 00000000000000000000000000000000 P e579829cd232471fbfef788138bcecad [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 20:28:00.870700 6688 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e579829cd232471fbfef788138bcecad [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:28:00.873267 6683 sys_catalog.cc:564] T 00000000000000000000000000000000 P e579829cd232471fbfef788138bcecad [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:28:00.877666 6688 raft_consensus.cc:686] T 00000000000000000000000000000000 P e579829cd232471fbfef788138bcecad [term 1 LEADER]: Becoming Leader. State: Replica: e579829cd232471fbfef788138bcecad, State: Running, Role: LEADER
I20260929 20:28:00.878474 6688 consensus_queue.cc:237] T 00000000000000000000000000000000 P e579829cd232471fbfef788138bcecad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e579829cd232471fbfef788138bcecad" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40481 } }
I20260929 20:28:00.882107 6689 sys_catalog.cc:455] T 00000000000000000000000000000000 P e579829cd232471fbfef788138bcecad [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e579829cd232471fbfef788138bcecad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e579829cd232471fbfef788138bcecad" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40481 } } }
I20260929 20:28:00.882583 6689 sys_catalog.cc:458] T 00000000000000000000000000000000 P e579829cd232471fbfef788138bcecad [sys.catalog]: This master's current role is: LEADER
I20260929 20:28:00.883237 6690 sys_catalog.cc:455] T 00000000000000000000000000000000 P e579829cd232471fbfef788138bcecad [sys.catalog]: SysCatalogTable state changed. Reason: New leader e579829cd232471fbfef788138bcecad. Latest consensus state: current_term: 1 leader_uuid: "e579829cd232471fbfef788138bcecad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e579829cd232471fbfef788138bcecad" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40481 } } }
I20260929 20:28:00.883703 6690 sys_catalog.cc:458] T 00000000000000000000000000000000 P e579829cd232471fbfef788138bcecad [sys.catalog]: This master's current role is: LEADER
I20260929 20:28:00.892026 6697 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:28:00.899358 6697 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:28:00.914678 6697 catalog_manager.cc:1347] Generated new cluster ID: a00788a85efa4a3fa20dd31909134483
I20260929 20:28:00.914790 6697 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:28:00.917323 6686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:28:00.894750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.118:43871 (local address 127.0.0.1:40481)
0929 20:28:00.894995 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0929 20:28:00.895005 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 20:28:00.895718 (+ 713us) server_negotiation.cc:407] Connection header received
0929 20:28:00.896061 (+ 343us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:28:00.896074 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:28:00.896187 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:28:00.896283 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:28:00.899149 (+ 2866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:28:00.900781 (+ 1632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:28:00.903313 (+ 2532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:28:00.904286 (+ 973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:28:00.904429 (+ 143us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:28:00.909182 (+ 4753us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:28:00.909193 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:28:00.909201 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:28:00.909256 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:28:00.913637 (+ 4381us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:28:00.914742 (+ 1105us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:28:00.914749 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:28:00.914753 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:28:00.914961 (+ 208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:28:00.915529 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:28:00.915535 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:28:00.915538 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:28:00.915986 (+ 448us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:28:00.916311 (+ 325us) server_negotiation.cc:1035] Waiting for connection context
0929 20:28:00.916741 (+ 430us) server_negotiation.cc:299] Negotiation successful
0929 20:28:00.916944 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260929 20:28:00.930696 6697 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 20:28:00.932719 6697 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:28:00.939189 6697 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e579829cd232471fbfef788138bcecad: Generated new TSK 0
I20260929 20:28:00.940263 6697 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 20:28:00.947151 17621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu pid 6616
I20260929 20:28:00.948256 6678 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:28:01.039079 17621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu with pid 6616
W20260929 20:28:01.059094 6601 connection.cc:535] client connection to 127.0.0.1:40481 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40481): Connection reset by peer (error 104)
2026-09-29T20:28:01Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (2030 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790713339161492-17621-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 20:28:01 dist-test-slave-zbnc krb5kdc[6716](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 20:28:01 dist-test-slave-zbnc krb5kdc[6716](info): set up 2 sockets
Sep 29 20:28:01 dist-test-slave-zbnc krb5kdc[6716](info): commencing operation
krb5kdc: starting...
W20260929 20:28:02.148810 17621 mini_kdc.cc:121] Time spent starting KDC: real 1.071s user 0.001s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790713339161492-17621-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790713339161492-17621-0/krb5kdc/test-user.keytab.
Sep 29 20:28:02 dist-test-slave-zbnc krb5kdc[6716](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713682, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T20:28:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T20:28:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790713339161492-17621-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790713339161492-17621-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 20:28:02.418521 17621 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
/tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790713339161492-17621-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790713339161492-17621-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790713339161492-17621-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790713339161492-17621-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41379
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790713339161492-17621-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:32957
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41379
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790713339161492-17621-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790713339161492-17621-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790713339161492-17621-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:28:02.754890 6729 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:28:02.755196 6729 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:28:02.755388 6729 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:28:02.762537 6729 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:28:02.762691 6729 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:28:02.762799 6729 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:28:02.762920 6729 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:28:02.763000 6729 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:28:02.773085 6729 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:32957
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790713339161492-17621-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790713339161492-17621-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41379
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790713339161492-17621-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41379
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790713339161492-17621-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790713339161492-17621-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:28:02.774428 6729 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:28:02.775774 6729 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:28:02.787613 6734 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:28:02.787701 6735 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:28:02.787642 6737 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:28:02.788712 6729 server_base.cc:1034] running on GCE node
I20260929 20:28:02.789409 6729 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:28:02.790721 6729 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:28:02.791923 6729 hybrid_clock.cc:648] HybridClock initialized: now 1790713682791876 us; error 42 us; skew 500 ppm
I20260929 20:28:02.792220 6729 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:28:02 dist-test-slave-zbnc krb5kdc[6716](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713682, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:28:02.795307 6729 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 20:28:02.796677 6729 webserver.cc:460] Webserver started at http://127.0.0.1:36867/ using document root <none> and password file <none>
I20260929 20:28:02.797312 6729 fs_manager.cc:359] Metadata directory not provided
I20260929 20:28:02.797416 6729 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:28:02.797670 6729 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:28:02.800606 6729 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790713339161492-17621-0/minicluster-data/master-0/data/instance:
uuid: "18a75d58661b482082766baa9f467a9c"
format_stamp: "Formatted at 2026-09-29 20:28:02 on dist-test-slave-zbnc"
server_key: "c444a7f93c3fb77be8c4b939c3dd708c"
server_key_iv: "b0c7d79bd5b6b9c7105b677483d0f09f"
server_key_version: "encryptionkey@0"
I20260929 20:28:02.801334 6729 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790713339161492-17621-0/minicluster-data/master-0/wal/instance:
uuid: "18a75d58661b482082766baa9f467a9c"
format_stamp: "Formatted at 2026-09-29 20:28:02 on dist-test-slave-zbnc"
server_key: "c444a7f93c3fb77be8c4b939c3dd708c"
server_key_iv: "b0c7d79bd5b6b9c7105b677483d0f09f"
server_key_version: "encryptionkey@0"
I20260929 20:28:02.806655 6729 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260929 20:28:02.810199 6744 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:28:02.811573 6729 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 20:28:02.811751 6729 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790713339161492-17621-0/minicluster-data/master-0/data,/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790713339161492-17621-0/minicluster-data/master-0/wal
uuid: "18a75d58661b482082766baa9f467a9c"
format_stamp: "Formatted at 2026-09-29 20:28:02 on dist-test-slave-zbnc"
server_key: "c444a7f93c3fb77be8c4b939c3dd708c"
server_key_iv: "b0c7d79bd5b6b9c7105b677483d0f09f"
server_key_version: "encryptionkey@0"
I20260929 20:28:02.811960 6729 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790713339161492-17621-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790713339161492-17621-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790713339161492-17621-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:28:02.839114 6729 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:28:02.844261 6729 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:28:02.844873 6729 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:28:02.873360 6729 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41379
I20260929 20:28:02.873462 6795 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41379 every 8 connection(s)
I20260929 20:28:02.875064 6729 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790713339161492-17621-0/minicluster-data/master-0/data/info.pb
I20260929 20:28:02.879877 6796 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:28:02.883608 17621 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu as pid 6729
I20260929 20:28:02.883802 17621 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790713339161492-17621-0/minicluster-data/master-0/wal/instance
I20260929 20:28:02.884402 17621 external_mini_cluster.cc:1428] Setting key ee6e8dd316159d51c2ee9313e9f75aa6
I20260929 20:28:02.895836 6796 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 18a75d58661b482082766baa9f467a9c: Bootstrap starting.
I20260929 20:28:02.900652 6796 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 18a75d58661b482082766baa9f467a9c: Neither blocks nor log segments found. Creating new log.
I20260929 20:28:02.902355 6796 log.cc:824] T 00000000000000000000000000000000 P 18a75d58661b482082766baa9f467a9c: Log is configured to *not* fsync() on all Append() calls
I20260929 20:28:02.902467 6799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:28:02.886594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.118:43213 (local address 127.0.0.1:41379)
0929 20:28:02.887515 (+ 921us) server_negotiation.cc:206] Beginning negotiation
0929 20:28:02.887533 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0929 20:28:02.887583 (+ 50us) server_negotiation.cc:407] Connection header received
0929 20:28:02.889907 (+ 2324us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:28:02.889941 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:28:02.890492 (+ 551us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:28:02.890974 (+ 482us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:28:02.891759 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:28:02.894060 (+ 2301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:28:02.895836 (+ 1776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:28:02.897045 (+ 1209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:28:02.898109 (+ 1064us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:28:02.899261 (+ 1152us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:28:02.899272 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:28:02.899290 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:28:02.900163 (+ 873us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:28:02.900271 (+ 108us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:28:02.900308 (+ 37us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:28:02.900512 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0929 20:28:02.900795 (+ 283us) server_negotiation.cc:299] Negotiation successful
0929 20:28:02.900998 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":663,"thread_start_us":393,"threads_started":1}
I20260929 20:28:02.906113 6796 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 18a75d58661b482082766baa9f467a9c: No bootstrap required, opened a new log
I20260929 20:28:02.907114 17621 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 20:28:02.917045 6796 raft_consensus.cc:348] T 00000000000000000000000000000000 P 18a75d58661b482082766baa9f467a9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18a75d58661b482082766baa9f467a9c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41379 } }
I20260929 20:28:02.917469 6796 raft_consensus.cc:374] T 00000000000000000000000000000000 P 18a75d58661b482082766baa9f467a9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:28:02.917606 6796 raft_consensus.cc:729] T 00000000000000000000000000000000 P 18a75d58661b482082766baa9f467a9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18a75d58661b482082766baa9f467a9c, State: Initialized, Role: FOLLOWER
I20260929 20:28:02.918200 6796 consensus_queue.cc:260] T 00000000000000000000000000000000 P 18a75d58661b482082766baa9f467a9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18a75d58661b482082766baa9f467a9c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41379 } }
I20260929 20:28:02.918470 6796 raft_consensus.cc:388] T 00000000000000000000000000000000 P 18a75d58661b482082766baa9f467a9c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 20:28:02.918677 6796 raft_consensus.cc:482] T 00000000000000000000000000000000 P 18a75d58661b482082766baa9f467a9c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 20:28:02.918884 6796 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 18a75d58661b482082766baa9f467a9c [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:28:02.919348 6799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:28:02.911964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.118:47079 (local address 127.0.0.1:41379)
0929 20:28:02.912130 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0929 20:28:02.912139 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 20:28:02.912924 (+ 785us) server_negotiation.cc:407] Connection header received
0929 20:28:02.913185 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:28:02.913194 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:28:02.913292 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:28:02.913407 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:28:02.914005 (+ 598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:28:02.914909 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:28:02.916642 (+ 1733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:28:02.917103 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:28:02.917249 (+ 146us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:28:02.918502 (+ 1253us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:28:02.918508 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:28:02.918512 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:28:02.918539 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:28:02.918636 (+ 97us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:28:02.918658 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:28:02.918802 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0929 20:28:02.919002 (+ 200us) server_negotiation.cc:299] Negotiation successful
0929 20:28:02.919055 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260929 20:28:02.921625 6796 raft_consensus.cc:504] T 00000000000000000000000000000000 P 18a75d58661b482082766baa9f467a9c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18a75d58661b482082766baa9f467a9c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41379 } }
I20260929 20:28:02.922223 6796 leader_election.cc:302] T 00000000000000000000000000000000 P 18a75d58661b482082766baa9f467a9c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 18a75d58661b482082766baa9f467a9c; no voters:
I20260929 20:28:02.922998 6796 leader_election.cc:288] T 00000000000000000000000000000000 P 18a75d58661b482082766baa9f467a9c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 20:28:02.923130 6806 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 18a75d58661b482082766baa9f467a9c [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:28:02.923924 6806 raft_consensus.cc:686] T 00000000000000000000000000000000 P 18a75d58661b482082766baa9f467a9c [term 1 LEADER]: Becoming Leader. State: Replica: 18a75d58661b482082766baa9f467a9c, State: Running, Role: LEADER
I20260929 20:28:02.924540 6806 consensus_queue.cc:237] T 00000000000000000000000000000000 P 18a75d58661b482082766baa9f467a9c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18a75d58661b482082766baa9f467a9c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41379 } }
I20260929 20:28:02.925421 6796 sys_catalog.cc:564] T 00000000000000000000000000000000 P 18a75d58661b482082766baa9f467a9c [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:28:02.927591 6808 sys_catalog.cc:455] T 00000000000000000000000000000000 P 18a75d58661b482082766baa9f467a9c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 18a75d58661b482082766baa9f467a9c. Latest consensus state: current_term: 1 leader_uuid: "18a75d58661b482082766baa9f467a9c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18a75d58661b482082766baa9f467a9c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41379 } } }
I20260929 20:28:02.927655 6807 sys_catalog.cc:455] T 00000000000000000000000000000000 P 18a75d58661b482082766baa9f467a9c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "18a75d58661b482082766baa9f467a9c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "18a75d58661b482082766baa9f467a9c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41379 } } }
I20260929 20:28:02.927985 6808 sys_catalog.cc:458] T 00000000000000000000000000000000 P 18a75d58661b482082766baa9f467a9c [sys.catalog]: This master's current role is: LEADER
I20260929 20:28:02.928020 6807 sys_catalog.cc:458] T 00000000000000000000000000000000 P 18a75d58661b482082766baa9f467a9c [sys.catalog]: This master's current role is: LEADER
I20260929 20:28:02.931129 6812 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:28:02.937659 6812 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:28:02.946666 6812 catalog_manager.cc:1347] Generated new cluster ID: 342897fb40954b07bc2d0ab8d1e8cb14
I20260929 20:28:02.946789 6812 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:28:02.966281 6812 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 20:28:02.968219 6812 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:28:02.977659 6812 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 18a75d58661b482082766baa9f467a9c: Generated new TSK 0
I20260929 20:28:02.978657 6812 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 20:28:02.991343 17621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu pid 6729
W20260929 20:28:02.992597 6791 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.118:43213
W20260929 20:28:02.993369 17621 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu pid 6729: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 20:28:02.993515 17621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu with pid 6729
W20260929 20:28:03.010727 6714 connection.cc:535] client connection to 127.0.0.1:41379 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41379): Connection reset by peer (error 104)
2026-09-29T20:28:03Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1955 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790713339161492-17621-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 20:28:03 dist-test-slave-zbnc krb5kdc[6827](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 20:28:03 dist-test-slave-zbnc krb5kdc[6827](info): set up 2 sockets
Sep 29 20:28:03 dist-test-slave-zbnc krb5kdc[6827](info): commencing operation
krb5kdc: starting...
W20260929 20:28:04.100716 17621 mini_kdc.cc:121] Time spent starting KDC: real 1.067s user 0.001s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790713339161492-17621-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790713339161492-17621-0/krb5kdc/test-user.keytab.
Sep 29 20:28:04 dist-test-slave-zbnc krb5kdc[6827](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713684, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T20:28:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T20:28:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790713339161492-17621-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790713339161492-17621-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260929 20:28:04.365888 17621 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
/tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790713339161492-17621-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790713339161492-17621-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790713339161492-17621-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790713339161492-17621-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42399
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790713339161492-17621-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43175
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42399
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790713339161492-17621-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790713339161492-17621-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790713339161492-17621-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:28:04.701344 6840 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:28:04.701627 6840 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:28:04.701725 6840 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:28:04.708868 6840 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:28:04.708981 6840 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:28:04.709038 6840 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:28:04.709091 6840 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:28:04.709134 6840 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:28:04.719069 6840 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43175
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790713339161492-17621-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790713339161492-17621-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42399
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790713339161492-17621-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42399
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790713339161492-17621-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790713339161492-17621-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:28:04.720412 6840 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:28:04.721695 6840 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:28:04.734122 6845 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:28:04.734014 6846 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:28:04.734082 6848 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:28:04.735174 6840 server_base.cc:1034] running on GCE node
I20260929 20:28:04.736024 6840 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:28:04.737530 6840 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:28:04.738801 6840 hybrid_clock.cc:648] HybridClock initialized: now 1790713684738744 us; error 50 us; skew 500 ppm
I20260929 20:28:04.739213 6840 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:28:04 dist-test-slave-zbnc krb5kdc[6827](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713684, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:28:04.748551 6840 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260929 20:28:04.750262 6840 webserver.cc:460] Webserver started at http://127.0.0.1:42179/ using document root <none> and password file <none>
I20260929 20:28:04.751099 6840 fs_manager.cc:359] Metadata directory not provided
I20260929 20:28:04.751255 6840 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:28:04.751598 6840 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:28:04.754799 6840 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790713339161492-17621-0/minicluster-data/master-0/data/instance:
uuid: "f46e2410581b4edaad5a0ddff422bbdf"
format_stamp: "Formatted at 2026-09-29 20:28:04 on dist-test-slave-zbnc"
server_key: "c84644e5d774f13950907ea71355ba5f"
server_key_iv: "18175d11c3524faf7dee83a7a496fa2e"
server_key_version: "encryptionkey@0"
I20260929 20:28:04.755700 6840 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790713339161492-17621-0/minicluster-data/master-0/wal/instance:
uuid: "f46e2410581b4edaad5a0ddff422bbdf"
format_stamp: "Formatted at 2026-09-29 20:28:04 on dist-test-slave-zbnc"
server_key: "c84644e5d774f13950907ea71355ba5f"
server_key_iv: "18175d11c3524faf7dee83a7a496fa2e"
server_key_version: "encryptionkey@0"
I20260929 20:28:04.761113 6840 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 20:28:04.764659 6855 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:28:04.766074 6840 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 20:28:04.766260 6840 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790713339161492-17621-0/minicluster-data/master-0/data,/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790713339161492-17621-0/minicluster-data/master-0/wal
uuid: "f46e2410581b4edaad5a0ddff422bbdf"
format_stamp: "Formatted at 2026-09-29 20:28:04 on dist-test-slave-zbnc"
server_key: "c84644e5d774f13950907ea71355ba5f"
server_key_iv: "18175d11c3524faf7dee83a7a496fa2e"
server_key_version: "encryptionkey@0"
I20260929 20:28:04.766476 6840 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790713339161492-17621-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790713339161492-17621-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790713339161492-17621-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:28:04.772159 6840 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:28:04.776533 6840 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:28:04.776889 6840 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:28:04.795588 6840 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42399
I20260929 20:28:04.795629 6906 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42399 every 8 connection(s)
I20260929 20:28:04.797025 6840 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790713339161492-17621-0/minicluster-data/master-0/data/info.pb
I20260929 20:28:04.800762 6907 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:28:04.801087 17621 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu as pid 6840
I20260929 20:28:04.801261 17621 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790713339161492-17621-0/minicluster-data/master-0/wal/instance
I20260929 20:28:04.801836 17621 external_mini_cluster.cc:1428] Setting key e26c6ecffd5edb137aba548d397f9075
I20260929 20:28:04.809245 6910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:28:04.803606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.118:56747 (local address 127.0.0.1:42399)
0929 20:28:04.804452 (+ 846us) server_negotiation.cc:206] Beginning negotiation
0929 20:28:04.804462 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 20:28:04.804502 (+ 40us) server_negotiation.cc:407] Connection header received
0929 20:28:04.805954 (+ 1452us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:28:04.805974 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:28:04.806307 (+ 333us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:28:04.806654 (+ 347us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:28:04.807716 (+ 1062us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:28:04.807722 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:28:04.807739 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:28:04.807753 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:28:04.807809 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:28:04.807828 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:28:04.808014 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0929 20:28:04.808131 (+ 117us) server_negotiation.cc:299] Negotiation successful
0929 20:28:04.808332 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":604,"thread_start_us":394,"threads_started":1}
I20260929 20:28:04.812835 17621 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 20:28:04.814744 6907 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f46e2410581b4edaad5a0ddff422bbdf: Bootstrap starting.
I20260929 20:28:04.818974 6907 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f46e2410581b4edaad5a0ddff422bbdf: Neither blocks nor log segments found. Creating new log.
I20260929 20:28:04.820427 6907 log.cc:824] T 00000000000000000000000000000000 P f46e2410581b4edaad5a0ddff422bbdf: Log is configured to *not* fsync() on all Append() calls
I20260929 20:28:04.820636 6910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:28:04.818049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.118:56959 (local address 127.0.0.1:42399)
0929 20:28:04.818208 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0929 20:28:04.818212 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:28:04.818556 (+ 344us) server_negotiation.cc:407] Connection header received
0929 20:28:04.818745 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:28:04.818749 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:28:04.818799 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:28:04.818972 (+ 173us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:28:04.819910 (+ 938us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:28:04.819914 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:28:04.819917 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:28:04.819932 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:28:04.819971 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:28:04.819986 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:28:04.820083 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0929 20:28:04.820229 (+ 146us) server_negotiation.cc:299] Negotiation successful
0929 20:28:04.820277 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260929 20:28:04.823468 6907 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f46e2410581b4edaad5a0ddff422bbdf: No bootstrap required, opened a new log
I20260929 20:28:04.831408 6907 raft_consensus.cc:348] T 00000000000000000000000000000000 P f46e2410581b4edaad5a0ddff422bbdf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f46e2410581b4edaad5a0ddff422bbdf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42399 } }
I20260929 20:28:04.831811 6907 raft_consensus.cc:374] T 00000000000000000000000000000000 P f46e2410581b4edaad5a0ddff422bbdf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:28:04.831923 6907 raft_consensus.cc:729] T 00000000000000000000000000000000 P f46e2410581b4edaad5a0ddff422bbdf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f46e2410581b4edaad5a0ddff422bbdf, State: Initialized, Role: FOLLOWER
I20260929 20:28:04.832513 6907 consensus_queue.cc:260] T 00000000000000000000000000000000 P f46e2410581b4edaad5a0ddff422bbdf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f46e2410581b4edaad5a0ddff422bbdf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42399 } }
I20260929 20:28:04.832762 6907 raft_consensus.cc:388] T 00000000000000000000000000000000 P f46e2410581b4edaad5a0ddff422bbdf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 20:28:04.832909 6907 raft_consensus.cc:482] T 00000000000000000000000000000000 P f46e2410581b4edaad5a0ddff422bbdf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 20:28:04.833076 6907 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f46e2410581b4edaad5a0ddff422bbdf [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:28:04.835306 6907 raft_consensus.cc:504] T 00000000000000000000000000000000 P f46e2410581b4edaad5a0ddff422bbdf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f46e2410581b4edaad5a0ddff422bbdf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42399 } }
I20260929 20:28:04.835925 6907 leader_election.cc:302] T 00000000000000000000000000000000 P f46e2410581b4edaad5a0ddff422bbdf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f46e2410581b4edaad5a0ddff422bbdf; no voters:
I20260929 20:28:04.837117 6907 leader_election.cc:288] T 00000000000000000000000000000000 P f46e2410581b4edaad5a0ddff422bbdf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 20:28:04.837246 6917 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f46e2410581b4edaad5a0ddff422bbdf [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:28:04.837915 6917 raft_consensus.cc:686] T 00000000000000000000000000000000 P f46e2410581b4edaad5a0ddff422bbdf [term 1 LEADER]: Becoming Leader. State: Replica: f46e2410581b4edaad5a0ddff422bbdf, State: Running, Role: LEADER
I20260929 20:28:04.838558 6917 consensus_queue.cc:237] T 00000000000000000000000000000000 P f46e2410581b4edaad5a0ddff422bbdf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f46e2410581b4edaad5a0ddff422bbdf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42399 } }
I20260929 20:28:04.839474 6907 sys_catalog.cc:564] T 00000000000000000000000000000000 P f46e2410581b4edaad5a0ddff422bbdf [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:28:04.841476 6919 sys_catalog.cc:455] T 00000000000000000000000000000000 P f46e2410581b4edaad5a0ddff422bbdf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f46e2410581b4edaad5a0ddff422bbdf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f46e2410581b4edaad5a0ddff422bbdf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42399 } } }
I20260929 20:28:04.841910 6919 sys_catalog.cc:458] T 00000000000000000000000000000000 P f46e2410581b4edaad5a0ddff422bbdf [sys.catalog]: This master's current role is: LEADER
I20260929 20:28:04.841900 6918 sys_catalog.cc:455] T 00000000000000000000000000000000 P f46e2410581b4edaad5a0ddff422bbdf [sys.catalog]: SysCatalogTable state changed. Reason: New leader f46e2410581b4edaad5a0ddff422bbdf. Latest consensus state: current_term: 1 leader_uuid: "f46e2410581b4edaad5a0ddff422bbdf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f46e2410581b4edaad5a0ddff422bbdf" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42399 } } }
I20260929 20:28:04.842199 6918 sys_catalog.cc:458] T 00000000000000000000000000000000 P f46e2410581b4edaad5a0ddff422bbdf [sys.catalog]: This master's current role is: LEADER
I20260929 20:28:04.844241 6925 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:28:04.850291 6925 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:28:04.859421 6925 catalog_manager.cc:1347] Generated new cluster ID: 8233c452c9e8442a8bd0bbcdcaa34957
I20260929 20:28:04.859540 6925 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:28:04.877461 6925 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 20:28:04.877904 6925 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:28:04.903515 6925 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f46e2410581b4edaad5a0ddff422bbdf: Generated new TSK 0
I20260929 20:28:04.904429 6925 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 20:28:04.912122 17621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu pid 6840
W20260929 20:28:04.913360 6902 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.118:56747
W20260929 20:28:04.914079 17621 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu pid 6840: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 20:28:04.914242 17621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu with pid 6840
2026-09-29T20:28:04Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1916 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790713339161492-17621-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 20:28:05 dist-test-slave-zbnc krb5kdc[6938](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 20:28:05 dist-test-slave-zbnc krb5kdc[6938](info): set up 2 sockets
Sep 29 20:28:05 dist-test-slave-zbnc krb5kdc[6938](info): commencing operation
krb5kdc: starting...
W20260929 20:28:06.016876 17621 mini_kdc.cc:121] Time spent starting KDC: real 1.066s user 0.002s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790713339161492-17621-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790713339161492-17621-0/krb5kdc/test-user.keytab.
Sep 29 20:28:06 dist-test-slave-zbnc krb5kdc[6938](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713686, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T20:28:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T20:28:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.17.53.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.53.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.53.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.126.keytab.
Entry for principal kudu/127.17.53.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.126.keytab.
I20260929 20:28:06.283038 17621 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
/tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790713339161492-17621-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790713339161492-17621-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790713339161492-17621-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790713339161492-17621-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.53.126:37559
--webserver_interface=127.17.53.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.126.keytab
--principal=kudu/127.17.53.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.53.84:34203
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.53.126:37559
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790713339161492-17621-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790713339161492-17621-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790713339161492-17621-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:28:06.619490 6951 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:28:06.619796 6951 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:28:06.620003 6951 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:28:06.631064 6951 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:28:06.631174 6951 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:28:06.631229 6951 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:28:06.631290 6951 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:28:06.631333 6951 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:28:06.641361 6951 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.53.84:34203
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790713339161492-17621-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790713339161492-17621-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.53.126:37559
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.126.keytab
--principal=kudu/127.17.53.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.53.126:37559
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790713339161492-17621-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.53.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790713339161492-17621-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:28:06.642617 6951 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:28:06.644022 6951 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:28:06.656517 6956 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:28:06.656448 6959 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:28:06.656486 6957 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:28:06.657584 6951 server_base.cc:1034] running on GCE node
I20260929 20:28:06.658342 6951 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:28:06.659870 6951 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:28:06.661100 6951 hybrid_clock.cc:648] HybridClock initialized: now 1790713686661046 us; error 37 us; skew 500 ppm
I20260929 20:28:06.661479 6951 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:28:06 dist-test-slave-zbnc krb5kdc[6938](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713686, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.53.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:28:06.664599 6951 init.cc:377] Logged in from keytab as kudu/127.17.53.126@KRBTEST.COM (short username kudu)
I20260929 20:28:06.666118 6951 webserver.cc:460] Webserver started at http://127.17.53.126:33133/ using document root <none> and password file <none>
I20260929 20:28:06.666795 6951 fs_manager.cc:359] Metadata directory not provided
I20260929 20:28:06.666986 6951 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:28:06.667289 6951 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:28:06.670212 6951 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790713339161492-17621-0/minicluster-data/master-0/data/instance:
uuid: "2dfa690de86743178c44dcb241cb075e"
format_stamp: "Formatted at 2026-09-29 20:28:06 on dist-test-slave-zbnc"
server_key: "c4131cb9e03a43ce08f07b4583d174da"
server_key_iv: "ac61ccb38a95cc388d6ccac1dd26ddb4"
server_key_version: "encryptionkey@0"
I20260929 20:28:06.670980 6951 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790713339161492-17621-0/minicluster-data/master-0/wal/instance:
uuid: "2dfa690de86743178c44dcb241cb075e"
format_stamp: "Formatted at 2026-09-29 20:28:06 on dist-test-slave-zbnc"
server_key: "c4131cb9e03a43ce08f07b4583d174da"
server_key_iv: "ac61ccb38a95cc388d6ccac1dd26ddb4"
server_key_version: "encryptionkey@0"
I20260929 20:28:06.675916 6951 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.001s
I20260929 20:28:06.679312 6966 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:28:06.680594 6951 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260929 20:28:06.680768 6951 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790713339161492-17621-0/minicluster-data/master-0/data,/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790713339161492-17621-0/minicluster-data/master-0/wal
uuid: "2dfa690de86743178c44dcb241cb075e"
format_stamp: "Formatted at 2026-09-29 20:28:06 on dist-test-slave-zbnc"
server_key: "c4131cb9e03a43ce08f07b4583d174da"
server_key_iv: "ac61ccb38a95cc388d6ccac1dd26ddb4"
server_key_version: "encryptionkey@0"
I20260929 20:28:06.680977 6951 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790713339161492-17621-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790713339161492-17621-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790713339161492-17621-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:28:06.699674 6951 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:28:06.703588 6951 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:28:06.703889 6951 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:28:06.722450 6951 rpc_server.cc:307] RPC server started. Bound to: 127.17.53.126:37559
I20260929 20:28:06.722469 7017 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.53.126:37559 every 8 connection(s)
I20260929 20:28:06.723881 6951 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790713339161492-17621-0/minicluster-data/master-0/data/info.pb
I20260929 20:28:06.727602 7018 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:28:06.728775 17621 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu as pid 6951
I20260929 20:28:06.728964 17621 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790713339161492-17621-0/minicluster-data/master-0/wal/instance
I20260929 20:28:06.729533 17621 external_mini_cluster.cc:1428] Setting key ee393693ca1069e422da516fa9fb5ef0
I20260929 20:28:06.740379 7018 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2dfa690de86743178c44dcb241cb075e: Bootstrap starting.
Sep 29 20:28:06 dist-test-slave-zbnc krb5kdc[6938](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713686, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.53.126@KRBTEST.COM
I20260929 20:28:06.743777 7018 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2dfa690de86743178c44dcb241cb075e: Neither blocks nor log segments found. Creating new log.
I20260929 20:28:06.744827 7018 log.cc:824] T 00000000000000000000000000000000 P 2dfa690de86743178c44dcb241cb075e: Log is configured to *not* fsync() on all Append() calls
I20260929 20:28:06.748138 7018 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2dfa690de86743178c44dcb241cb075e: No bootstrap required, opened a new log
I20260929 20:28:06.752930 7021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:28:06.731307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53787 (local address 127.17.53.126:37559)
0929 20:28:06.732061 (+ 754us) server_negotiation.cc:206] Beginning negotiation
0929 20:28:06.732072 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 20:28:06.732110 (+ 38us) server_negotiation.cc:407] Connection header received
0929 20:28:06.734207 (+ 2097us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:28:06.734229 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:28:06.734630 (+ 401us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:28:06.735100 (+ 470us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:28:06.736547 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:28:06.737863 (+ 1316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:28:06.739204 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:28:06.739910 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:28:06.740111 (+ 201us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:28:06.744502 (+ 4391us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:28:06.744510 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:28:06.744525 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:28:06.744569 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:28:06.748086 (+ 3517us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:28:06.749092 (+ 1006us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:28:06.749101 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:28:06.749110 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:28:06.749315 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:28:06.750085 (+ 770us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:28:06.750092 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:28:06.750096 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:28:06.750652 (+ 556us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:28:06.751087 (+ 435us) server_negotiation.cc:1035] Waiting for connection context
0929 20:28:06.751544 (+ 457us) server_negotiation.cc:299] Negotiation successful
0929 20:28:06.751903 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":539,"thread_start_us":298,"threads_started":1}
I20260929 20:28:06.757061 17621 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 20:28:06.757401 7018 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2dfa690de86743178c44dcb241cb075e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2dfa690de86743178c44dcb241cb075e" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 37559 } }
I20260929 20:28:06.757767 7018 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2dfa690de86743178c44dcb241cb075e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:28:06.757874 7018 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2dfa690de86743178c44dcb241cb075e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2dfa690de86743178c44dcb241cb075e, State: Initialized, Role: FOLLOWER
I20260929 20:28:06.758409 7018 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2dfa690de86743178c44dcb241cb075e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2dfa690de86743178c44dcb241cb075e" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 37559 } }
I20260929 20:28:06.758652 7018 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2dfa690de86743178c44dcb241cb075e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 20:28:06.758792 7018 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2dfa690de86743178c44dcb241cb075e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 20:28:06.758973 7018 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2dfa690de86743178c44dcb241cb075e [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:28:06.761112 7018 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2dfa690de86743178c44dcb241cb075e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2dfa690de86743178c44dcb241cb075e" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 37559 } }
I20260929 20:28:06.761615 7018 leader_election.cc:302] T 00000000000000000000000000000000 P 2dfa690de86743178c44dcb241cb075e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2dfa690de86743178c44dcb241cb075e; no voters:
I20260929 20:28:06.762293 7018 leader_election.cc:288] T 00000000000000000000000000000000 P 2dfa690de86743178c44dcb241cb075e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 20:28:06.762389 7028 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2dfa690de86743178c44dcb241cb075e [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:28:06.764868 7018 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2dfa690de86743178c44dcb241cb075e [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:28:06.769356 7028 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2dfa690de86743178c44dcb241cb075e [term 1 LEADER]: Becoming Leader. State: Replica: 2dfa690de86743178c44dcb241cb075e, State: Running, Role: LEADER
I20260929 20:28:06.770116 7028 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2dfa690de86743178c44dcb241cb075e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2dfa690de86743178c44dcb241cb075e" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 37559 } }
I20260929 20:28:06.773523 7029 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2dfa690de86743178c44dcb241cb075e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2dfa690de86743178c44dcb241cb075e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2dfa690de86743178c44dcb241cb075e" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 37559 } } }
I20260929 20:28:06.773624 7030 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2dfa690de86743178c44dcb241cb075e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2dfa690de86743178c44dcb241cb075e. Latest consensus state: current_term: 1 leader_uuid: "2dfa690de86743178c44dcb241cb075e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2dfa690de86743178c44dcb241cb075e" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 37559 } } }
I20260929 20:28:06.773939 7029 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2dfa690de86743178c44dcb241cb075e [sys.catalog]: This master's current role is: LEADER
I20260929 20:28:06.774003 7030 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2dfa690de86743178c44dcb241cb075e [sys.catalog]: This master's current role is: LEADER
I20260929 20:28:06.774909 7035 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:28:06.778662 7021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:28:06.761768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34191 (local address 127.17.53.126:37559)
0929 20:28:06.761942 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0929 20:28:06.761948 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:28:06.762510 (+ 562us) server_negotiation.cc:407] Connection header received
0929 20:28:06.762798 (+ 288us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:28:06.762805 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:28:06.762924 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:28:06.763017 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:28:06.764657 (+ 1640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:28:06.765543 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:28:06.766944 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:28:06.767427 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:28:06.767571 (+ 144us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:28:06.770092 (+ 2521us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:28:06.770097 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:28:06.770104 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:28:06.770133 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:28:06.773843 (+ 3710us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:28:06.774894 (+ 1051us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:28:06.774899 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:28:06.774902 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:28:06.775062 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:28:06.775690 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:28:06.775697 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:28:06.775701 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:28:06.776141 (+ 440us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:28:06.776395 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0929 20:28:06.777916 (+ 1521us) server_negotiation.cc:299] Negotiation successful
0929 20:28:06.778164 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260929 20:28:06.787758 7035 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:28:06.796759 7035 catalog_manager.cc:1347] Generated new cluster ID: affbe2599afc49f28bd2c70d7604ae51
I20260929 20:28:06.796881 7035 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:28:06.810117 7035 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 20:28:06.812011 7035 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:28:06.822531 7035 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2dfa690de86743178c44dcb241cb075e: Generated new TSK 0
I20260929 20:28:06.823642 7035 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 20:28:06.833505 17621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu pid 6951
I20260929 20:28:06.834786 7013 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:28:06.925631 17621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu with pid 6951
W20260929 20:28:06.942648 6936 connection.cc:535] client connection to 127.17.53.126:37559 recv error: Network error: failed to read from TLS socket (remote: 127.17.53.126:37559): Connection reset by peer (error 104)
2026-09-29T20:28:06Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (2011 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790713339161492-17621-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 20:28:07 dist-test-slave-zbnc krb5kdc[7050](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 20:28:07 dist-test-slave-zbnc krb5kdc[7050](info): set up 2 sockets
Sep 29 20:28:07 dist-test-slave-zbnc krb5kdc[7050](info): commencing operation
krb5kdc: starting...
W20260929 20:28:08.028723 17621 mini_kdc.cc:121] Time spent starting KDC: real 1.067s user 0.001s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790713339161492-17621-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790713339161492-17621-0/krb5kdc/test-user.keytab.
Sep 29 20:28:08 dist-test-slave-zbnc krb5kdc[7050](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713688, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T20:28:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T20:28:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.17.53.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.53.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.53.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.126.keytab.
Entry for principal kudu/127.17.53.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.126.keytab.
I20260929 20:28:08.292006 17621 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
/tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790713339161492-17621-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790713339161492-17621-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790713339161492-17621-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790713339161492-17621-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.53.126:42951
--webserver_interface=127.17.53.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.126.keytab
--principal=kudu/127.17.53.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.53.84:42699
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.53.126:42951
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790713339161492-17621-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790713339161492-17621-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790713339161492-17621-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:28:08.626574 7063 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:28:08.626978 7063 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:28:08.627139 7063 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:28:08.634600 7063 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:28:08.634766 7063 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:28:08.634909 7063 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:28:08.635017 7063 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:28:08.635090 7063 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:28:08.645224 7063 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.53.84:42699
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790713339161492-17621-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790713339161492-17621-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.53.126:42951
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.126.keytab
--principal=kudu/127.17.53.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.53.126:42951
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790713339161492-17621-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.53.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790713339161492-17621-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:28:08.646617 7063 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:28:08.648032 7063 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:28:08.659961 7071 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:28:08.660159 7068 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:28:08.660130 7069 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:28:08.661141 7063 server_base.cc:1034] running on GCE node
I20260929 20:28:08.661952 7063 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:28:08.663440 7063 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:28:08.664642 7063 hybrid_clock.cc:648] HybridClock initialized: now 1790713688664602 us; error 32 us; skew 500 ppm
I20260929 20:28:08.665012 7063 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:28:08 dist-test-slave-zbnc krb5kdc[7050](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713688, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.53.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:28:08.668288 7063 init.cc:377] Logged in from keytab as kudu/127.17.53.126@KRBTEST.COM (short username kudu)
I20260929 20:28:08.669847 7063 webserver.cc:460] Webserver started at http://127.17.53.126:32867/ using document root <none> and password file <none>
I20260929 20:28:08.670588 7063 fs_manager.cc:359] Metadata directory not provided
I20260929 20:28:08.670739 7063 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:28:08.671137 7063 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:28:08.674252 7063 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790713339161492-17621-0/minicluster-data/master-0/data/instance:
uuid: "fc9df38aafa14c8a86ee3bbbfe134aa4"
format_stamp: "Formatted at 2026-09-29 20:28:08 on dist-test-slave-zbnc"
server_key: "5884150c6a472d8904aef82a3a70065f"
server_key_iv: "5724025bfc269372654c72382cf72bc6"
server_key_version: "encryptionkey@0"
I20260929 20:28:08.675139 7063 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790713339161492-17621-0/minicluster-data/master-0/wal/instance:
uuid: "fc9df38aafa14c8a86ee3bbbfe134aa4"
format_stamp: "Formatted at 2026-09-29 20:28:08 on dist-test-slave-zbnc"
server_key: "5884150c6a472d8904aef82a3a70065f"
server_key_iv: "5724025bfc269372654c72382cf72bc6"
server_key_version: "encryptionkey@0"
I20260929 20:28:08.680785 7063 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 20:28:08.684506 7078 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:28:08.685977 7063 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 20:28:08.686178 7063 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790713339161492-17621-0/minicluster-data/master-0/data,/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790713339161492-17621-0/minicluster-data/master-0/wal
uuid: "fc9df38aafa14c8a86ee3bbbfe134aa4"
format_stamp: "Formatted at 2026-09-29 20:28:08 on dist-test-slave-zbnc"
server_key: "5884150c6a472d8904aef82a3a70065f"
server_key_iv: "5724025bfc269372654c72382cf72bc6"
server_key_version: "encryptionkey@0"
I20260929 20:28:08.686426 7063 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790713339161492-17621-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790713339161492-17621-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790713339161492-17621-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:28:08.710472 7063 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:28:08.714824 7063 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:28:08.715189 7063 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:28:08.734642 7063 rpc_server.cc:307] RPC server started. Bound to: 127.17.53.126:42951
I20260929 20:28:08.734665 7129 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.53.126:42951 every 8 connection(s)
I20260929 20:28:08.736136 7063 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790713339161492-17621-0/minicluster-data/master-0/data/info.pb
I20260929 20:28:08.737504 17621 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu as pid 7063
I20260929 20:28:08.737686 17621 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790713339161492-17621-0/minicluster-data/master-0/wal/instance
I20260929 20:28:08.738278 17621 external_mini_cluster.cc:1428] Setting key 72ae3f26406d07a32e84d200105a2c75
I20260929 20:28:08.741009 7130 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:28:08.752090 7133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:28:08.740190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41243 (local address 127.17.53.126:42951)
0929 20:28:08.740989 (+ 799us) server_negotiation.cc:206] Beginning negotiation
0929 20:28:08.741006 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0929 20:28:08.741049 (+ 43us) server_negotiation.cc:407] Connection header received
0929 20:28:08.742740 (+ 1691us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:28:08.742762 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:28:08.743240 (+ 478us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:28:08.743620 (+ 380us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:28:08.744238 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:28:08.745564 (+ 1326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:28:08.747507 (+ 1943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:28:08.748170 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:28:08.748306 (+ 136us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:28:08.749675 (+ 1369us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:28:08.749682 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:28:08.749696 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:28:08.750267 (+ 571us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:28:08.750353 (+ 86us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:28:08.750381 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:28:08.750503 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0929 20:28:08.750807 (+ 304us) server_negotiation.cc:299] Negotiation successful
0929 20:28:08.751078 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":583,"thread_start_us":368,"threads_started":1}
I20260929 20:28:08.753521 7130 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc9df38aafa14c8a86ee3bbbfe134aa4: Bootstrap starting.
I20260929 20:28:08.756040 17621 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 20:28:08.757010 7130 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fc9df38aafa14c8a86ee3bbbfe134aa4: Neither blocks nor log segments found. Creating new log.
I20260929 20:28:08.758216 7130 log.cc:824] T 00000000000000000000000000000000 P fc9df38aafa14c8a86ee3bbbfe134aa4: Log is configured to *not* fsync() on all Append() calls
I20260929 20:28:08.761492 7130 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fc9df38aafa14c8a86ee3bbbfe134aa4: No bootstrap required, opened a new log
I20260929 20:28:08.768401 7133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:28:08.761358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49405 (local address 127.17.53.126:42951)
0929 20:28:08.761598 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0929 20:28:08.761604 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:28:08.761965 (+ 361us) server_negotiation.cc:407] Connection header received
0929 20:28:08.762188 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:28:08.762192 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:28:08.762250 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:28:08.762331 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:28:08.762954 (+ 623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:28:08.763729 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:28:08.765694 (+ 1965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:28:08.766159 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:28:08.766256 (+ 97us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:28:08.767570 (+ 1314us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:28:08.767574 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:28:08.767577 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:28:08.767595 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:28:08.767635 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:28:08.767653 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:28:08.767733 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0929 20:28:08.767961 (+ 228us) server_negotiation.cc:299] Negotiation successful
0929 20:28:08.768037 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260929 20:28:08.771679 7130 raft_consensus.cc:348] T 00000000000000000000000000000000 P fc9df38aafa14c8a86ee3bbbfe134aa4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc9df38aafa14c8a86ee3bbbfe134aa4" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 42951 } }
I20260929 20:28:08.772085 7130 raft_consensus.cc:374] T 00000000000000000000000000000000 P fc9df38aafa14c8a86ee3bbbfe134aa4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:28:08.772218 7130 raft_consensus.cc:729] T 00000000000000000000000000000000 P fc9df38aafa14c8a86ee3bbbfe134aa4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc9df38aafa14c8a86ee3bbbfe134aa4, State: Initialized, Role: FOLLOWER
I20260929 20:28:08.772962 7130 consensus_queue.cc:260] T 00000000000000000000000000000000 P fc9df38aafa14c8a86ee3bbbfe134aa4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc9df38aafa14c8a86ee3bbbfe134aa4" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 42951 } }
I20260929 20:28:08.773253 7130 raft_consensus.cc:388] T 00000000000000000000000000000000 P fc9df38aafa14c8a86ee3bbbfe134aa4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 20:28:08.773418 7130 raft_consensus.cc:482] T 00000000000000000000000000000000 P fc9df38aafa14c8a86ee3bbbfe134aa4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 20:28:08.773617 7130 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fc9df38aafa14c8a86ee3bbbfe134aa4 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:28:08.775936 7130 raft_consensus.cc:504] T 00000000000000000000000000000000 P fc9df38aafa14c8a86ee3bbbfe134aa4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc9df38aafa14c8a86ee3bbbfe134aa4" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 42951 } }
I20260929 20:28:08.776471 7130 leader_election.cc:302] T 00000000000000000000000000000000 P fc9df38aafa14c8a86ee3bbbfe134aa4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fc9df38aafa14c8a86ee3bbbfe134aa4; no voters:
I20260929 20:28:08.777065 7130 leader_election.cc:288] T 00000000000000000000000000000000 P fc9df38aafa14c8a86ee3bbbfe134aa4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 20:28:08.777220 7140 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fc9df38aafa14c8a86ee3bbbfe134aa4 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:28:08.778040 7140 raft_consensus.cc:686] T 00000000000000000000000000000000 P fc9df38aafa14c8a86ee3bbbfe134aa4 [term 1 LEADER]: Becoming Leader. State: Replica: fc9df38aafa14c8a86ee3bbbfe134aa4, State: Running, Role: LEADER
I20260929 20:28:08.778568 7140 consensus_queue.cc:237] T 00000000000000000000000000000000 P fc9df38aafa14c8a86ee3bbbfe134aa4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc9df38aafa14c8a86ee3bbbfe134aa4" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 42951 } }
I20260929 20:28:08.779381 7130 sys_catalog.cc:564] T 00000000000000000000000000000000 P fc9df38aafa14c8a86ee3bbbfe134aa4 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:28:08.781812 7142 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc9df38aafa14c8a86ee3bbbfe134aa4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fc9df38aafa14c8a86ee3bbbfe134aa4. Latest consensus state: current_term: 1 leader_uuid: "fc9df38aafa14c8a86ee3bbbfe134aa4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc9df38aafa14c8a86ee3bbbfe134aa4" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 42951 } } }
I20260929 20:28:08.781857 7141 sys_catalog.cc:455] T 00000000000000000000000000000000 P fc9df38aafa14c8a86ee3bbbfe134aa4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fc9df38aafa14c8a86ee3bbbfe134aa4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fc9df38aafa14c8a86ee3bbbfe134aa4" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 42951 } } }
I20260929 20:28:08.782354 7142 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc9df38aafa14c8a86ee3bbbfe134aa4 [sys.catalog]: This master's current role is: LEADER
I20260929 20:28:08.782384 7141 sys_catalog.cc:458] T 00000000000000000000000000000000 P fc9df38aafa14c8a86ee3bbbfe134aa4 [sys.catalog]: This master's current role is: LEADER
I20260929 20:28:08.787878 7149 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:28:08.793113 7149 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:28:08.801693 7149 catalog_manager.cc:1347] Generated new cluster ID: 83cfa9a954c74647bddafa4599534901
I20260929 20:28:08.801869 7149 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:28:08.813786 7149 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 20:28:08.815147 7149 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:28:08.823419 7149 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fc9df38aafa14c8a86ee3bbbfe134aa4: Generated new TSK 0
I20260929 20:28:08.824517 7149 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 20:28:08.839428 17621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu pid 7063
W20260929 20:28:08.840592 7125 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:41243
W20260929 20:28:08.841346 17621 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu pid 7063: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 20:28:08.841511 17621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu with pid 7063
W20260929 20:28:08.860754 7048 connection.cc:535] client connection to 127.17.53.126:42951 recv error: Network error: failed to read from TLS socket (remote: 127.17.53.126:42951): Connection reset by peer (error 104)
2026-09-29T20:28:08Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1917 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790713339161492-17621-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 20:28:08 dist-test-slave-zbnc krb5kdc[7161](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 20:28:08 dist-test-slave-zbnc krb5kdc[7161](info): set up 2 sockets
Sep 29 20:28:08 dist-test-slave-zbnc krb5kdc[7161](info): commencing operation
krb5kdc: starting...
W20260929 20:28:09.956722 17621 mini_kdc.cc:121] Time spent starting KDC: real 1.077s user 0.005s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790713339161492-17621-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790713339161492-17621-0/krb5kdc/test-user.keytab.
Sep 29 20:28:10 dist-test-slave-zbnc krb5kdc[7161](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713690, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T20:28:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T20:28:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.17.53.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.53.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.53.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.126.keytab.
Entry for principal kudu/127.17.53.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.126.keytab.
I20260929 20:28:10.222098 17621 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
/tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790713339161492-17621-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790713339161492-17621-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790713339161492-17621-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790713339161492-17621-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.53.126:45093
--webserver_interface=127.17.53.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.126.keytab
--principal=kudu/127.17.53.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.53.84:35965
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.53.126:45093
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790713339161492-17621-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790713339161492-17621-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790713339161492-17621-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:28:10.554661 7174 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:28:10.555042 7174 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:28:10.555147 7174 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:28:10.562331 7174 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:28:10.562443 7174 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:28:10.562489 7174 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:28:10.562544 7174 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:28:10.562589 7174 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:28:10.572347 7174 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.53.84:35965
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790713339161492-17621-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790713339161492-17621-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.53.126:45093
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.126.keytab
--principal=kudu/127.17.53.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.53.126:45093
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790713339161492-17621-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.53.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790713339161492-17621-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:28:10.573709 7174 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:28:10.575039 7174 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:28:10.586846 7179 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:28:10.586944 7182 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:28:10.586910 7180 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:28:10.587769 7174 server_base.cc:1034] running on GCE node
I20260929 20:28:10.588536 7174 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:28:10.589865 7174 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:28:10.591102 7174 hybrid_clock.cc:648] HybridClock initialized: now 1790713690591056 us; error 32 us; skew 500 ppm
I20260929 20:28:10.591455 7174 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:28:10 dist-test-slave-zbnc krb5kdc[7161](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713690, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.53.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:28:10.598757 7174 init.cc:377] Logged in from keytab as kudu/127.17.53.126@KRBTEST.COM (short username kudu)
I20260929 20:28:10.600255 7174 webserver.cc:460] Webserver started at http://127.17.53.126:40795/ using document root <none> and password file <none>
I20260929 20:28:10.600901 7174 fs_manager.cc:359] Metadata directory not provided
I20260929 20:28:10.601006 7174 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:28:10.601286 7174 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:28:10.604318 7174 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790713339161492-17621-0/minicluster-data/master-0/data/instance:
uuid: "66614a9bd2a94defbe63e10efb37db8d"
format_stamp: "Formatted at 2026-09-29 20:28:10 on dist-test-slave-zbnc"
server_key: "f3837bdf4a128036ef08d380d06066ad"
server_key_iv: "3e8651a7a932ce97ae809e9fb230453e"
server_key_version: "encryptionkey@0"
I20260929 20:28:10.605021 7174 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790713339161492-17621-0/minicluster-data/master-0/wal/instance:
uuid: "66614a9bd2a94defbe63e10efb37db8d"
format_stamp: "Formatted at 2026-09-29 20:28:10 on dist-test-slave-zbnc"
server_key: "f3837bdf4a128036ef08d380d06066ad"
server_key_iv: "3e8651a7a932ce97ae809e9fb230453e"
server_key_version: "encryptionkey@0"
I20260929 20:28:10.610117 7174 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260929 20:28:10.613530 7189 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:28:10.615208 7174 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260929 20:28:10.615366 7174 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790713339161492-17621-0/minicluster-data/master-0/data,/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790713339161492-17621-0/minicluster-data/master-0/wal
uuid: "66614a9bd2a94defbe63e10efb37db8d"
format_stamp: "Formatted at 2026-09-29 20:28:10 on dist-test-slave-zbnc"
server_key: "f3837bdf4a128036ef08d380d06066ad"
server_key_iv: "3e8651a7a932ce97ae809e9fb230453e"
server_key_version: "encryptionkey@0"
I20260929 20:28:10.615543 7174 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790713339161492-17621-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790713339161492-17621-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790713339161492-17621-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:28:10.621445 7174 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:28:10.625756 7174 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:28:10.626073 7174 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:28:10.645592 7174 rpc_server.cc:307] RPC server started. Bound to: 127.17.53.126:45093
I20260929 20:28:10.645627 7240 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.53.126:45093 every 8 connection(s)
I20260929 20:28:10.647042 7174 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790713339161492-17621-0/minicluster-data/master-0/data/info.pb
I20260929 20:28:10.650758 7241 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:28:10.656750 17621 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu as pid 7174
I20260929 20:28:10.656935 17621 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790713339161492-17621-0/minicluster-data/master-0/wal/instance
I20260929 20:28:10.657415 17621 external_mini_cluster.cc:1428] Setting key d9a951f56038aa1cc522f9aafa4a4c87
I20260929 20:28:10.662369 7241 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 66614a9bd2a94defbe63e10efb37db8d: Bootstrap starting.
I20260929 20:28:10.666373 7241 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 66614a9bd2a94defbe63e10efb37db8d: Neither blocks nor log segments found. Creating new log.
I20260929 20:28:10.666436 7244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:28:10.659411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33273 (local address 127.17.53.126:45093)
0929 20:28:10.660229 (+ 818us) server_negotiation.cc:206] Beginning negotiation
0929 20:28:10.660244 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 20:28:10.660293 (+ 49us) server_negotiation.cc:407] Connection header received
0929 20:28:10.662603 (+ 2310us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:28:10.662631 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:28:10.663075 (+ 444us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:28:10.663495 (+ 420us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:28:10.664517 (+ 1022us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:28:10.664525 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:28:10.664541 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:28:10.664568 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:28:10.664634 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:28:10.664673 (+ 39us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:28:10.664876 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0929 20:28:10.665001 (+ 125us) server_negotiation.cc:299] Negotiation successful
0929 20:28:10.665174 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":566,"thread_start_us":343,"threads_started":1}
I20260929 20:28:10.667593 7241 log.cc:824] T 00000000000000000000000000000000 P 66614a9bd2a94defbe63e10efb37db8d: Log is configured to *not* fsync() on all Append() calls
I20260929 20:28:10.670367 17621 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 20:28:10.670806 7241 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 66614a9bd2a94defbe63e10efb37db8d: No bootstrap required, opened a new log
I20260929 20:28:10.677853 7244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:28:10.675317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34187 (local address 127.17.53.126:45093)
0929 20:28:10.675524 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0929 20:28:10.675532 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 20:28:10.675976 (+ 444us) server_negotiation.cc:407] Connection header received
0929 20:28:10.676146 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:28:10.676150 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:28:10.676196 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:28:10.676276 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:28:10.677208 (+ 932us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:28:10.677212 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:28:10.677216 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:28:10.677230 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:28:10.677268 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:28:10.677282 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:28:10.677380 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0929 20:28:10.677508 (+ 128us) server_negotiation.cc:299] Negotiation successful
0929 20:28:10.677554 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 20:28:10.680871 7241 raft_consensus.cc:348] T 00000000000000000000000000000000 P 66614a9bd2a94defbe63e10efb37db8d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66614a9bd2a94defbe63e10efb37db8d" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 45093 } }
I20260929 20:28:10.681308 7241 raft_consensus.cc:374] T 00000000000000000000000000000000 P 66614a9bd2a94defbe63e10efb37db8d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:28:10.681422 7241 raft_consensus.cc:729] T 00000000000000000000000000000000 P 66614a9bd2a94defbe63e10efb37db8d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 66614a9bd2a94defbe63e10efb37db8d, State: Initialized, Role: FOLLOWER
I20260929 20:28:10.681974 7241 consensus_queue.cc:260] T 00000000000000000000000000000000 P 66614a9bd2a94defbe63e10efb37db8d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66614a9bd2a94defbe63e10efb37db8d" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 45093 } }
I20260929 20:28:10.682207 7241 raft_consensus.cc:388] T 00000000000000000000000000000000 P 66614a9bd2a94defbe63e10efb37db8d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 20:28:10.682367 7241 raft_consensus.cc:482] T 00000000000000000000000000000000 P 66614a9bd2a94defbe63e10efb37db8d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 20:28:10.682536 7241 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 66614a9bd2a94defbe63e10efb37db8d [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:28:10.684736 7241 raft_consensus.cc:504] T 00000000000000000000000000000000 P 66614a9bd2a94defbe63e10efb37db8d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66614a9bd2a94defbe63e10efb37db8d" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 45093 } }
I20260929 20:28:10.685256 7241 leader_election.cc:302] T 00000000000000000000000000000000 P 66614a9bd2a94defbe63e10efb37db8d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 66614a9bd2a94defbe63e10efb37db8d; no voters:
I20260929 20:28:10.685863 7241 leader_election.cc:288] T 00000000000000000000000000000000 P 66614a9bd2a94defbe63e10efb37db8d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 20:28:10.686131 7251 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 66614a9bd2a94defbe63e10efb37db8d [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:28:10.686811 7251 raft_consensus.cc:686] T 00000000000000000000000000000000 P 66614a9bd2a94defbe63e10efb37db8d [term 1 LEADER]: Becoming Leader. State: Replica: 66614a9bd2a94defbe63e10efb37db8d, State: Running, Role: LEADER
I20260929 20:28:10.687407 7251 consensus_queue.cc:237] T 00000000000000000000000000000000 P 66614a9bd2a94defbe63e10efb37db8d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66614a9bd2a94defbe63e10efb37db8d" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 45093 } }
I20260929 20:28:10.687922 7241 sys_catalog.cc:564] T 00000000000000000000000000000000 P 66614a9bd2a94defbe63e10efb37db8d [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:28:10.690040 7253 sys_catalog.cc:455] T 00000000000000000000000000000000 P 66614a9bd2a94defbe63e10efb37db8d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 66614a9bd2a94defbe63e10efb37db8d. Latest consensus state: current_term: 1 leader_uuid: "66614a9bd2a94defbe63e10efb37db8d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66614a9bd2a94defbe63e10efb37db8d" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 45093 } } }
I20260929 20:28:10.690445 7253 sys_catalog.cc:458] T 00000000000000000000000000000000 P 66614a9bd2a94defbe63e10efb37db8d [sys.catalog]: This master's current role is: LEADER
I20260929 20:28:10.691027 7252 sys_catalog.cc:455] T 00000000000000000000000000000000 P 66614a9bd2a94defbe63e10efb37db8d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "66614a9bd2a94defbe63e10efb37db8d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "66614a9bd2a94defbe63e10efb37db8d" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 45093 } } }
I20260929 20:28:10.691355 7252 sys_catalog.cc:458] T 00000000000000000000000000000000 P 66614a9bd2a94defbe63e10efb37db8d [sys.catalog]: This master's current role is: LEADER
I20260929 20:28:10.691497 7258 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:28:10.705440 7258 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:28:10.712082 7269 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260929 20:28:10.716013 7258 catalog_manager.cc:1347] Generated new cluster ID: 6a5e291661144a218a2618fca8bc6f32
I20260929 20:28:10.716143 7258 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:28:10.740264 7258 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 20:28:10.740798 7258 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:28:10.758782 7258 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 66614a9bd2a94defbe63e10efb37db8d: Generated new TSK 0
I20260929 20:28:10.759779 7258 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 20:28:10.771133 17621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu pid 7174
W20260929 20:28:10.772250 7236 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:33273
W20260929 20:28:10.772985 17621 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu pid 7174: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 20:28:10.773118 17621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu with pid 7174
2026-09-29T20:28:10Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1929 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790713339161492-17621-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 20:28:10 dist-test-slave-zbnc krb5kdc[7273](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 20:28:10 dist-test-slave-zbnc krb5kdc[7273](info): set up 2 sockets
Sep 29 20:28:10 dist-test-slave-zbnc krb5kdc[7273](info): commencing operation
krb5kdc: starting...
W20260929 20:28:11.900677 17621 mini_kdc.cc:121] Time spent starting KDC: real 1.091s user 0.007s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790713339161492-17621-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790713339161492-17621-0/krb5kdc/test-user.keytab.
Sep 29 20:28:12 dist-test-slave-zbnc krb5kdc[7273](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713692, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T20:28:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T20:28:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.17.53.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.53.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.53.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.126.keytab.
Entry for principal kudu/127.17.53.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.126.keytab.
I20260929 20:28:12.166134 17621 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
/tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790713339161492-17621-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790713339161492-17621-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790713339161492-17621-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790713339161492-17621-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.53.126:37885
--webserver_interface=127.17.53.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.126.keytab
--principal=kudu/127.17.53.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.53.84:42131
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.53.126:37885
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790713339161492-17621-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790713339161492-17621-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790713339161492-17621-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:28:12.505343 7286 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:28:12.505724 7286 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:28:12.505914 7286 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:28:12.513970 7286 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:28:12.514168 7286 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:28:12.514288 7286 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:28:12.514390 7286 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:28:12.514464 7286 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:28:12.525444 7286 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.53.84:42131
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790713339161492-17621-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790713339161492-17621-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.53.126:37885
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.126.keytab
--principal=kudu/127.17.53.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.53.126:37885
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790713339161492-17621-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.53.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790713339161492-17621-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:28:12.526968 7286 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:28:12.528455 7286 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:28:12.541954 7294 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:28:12.542062 7292 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:28:12.542085 7291 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:28:12.543005 7286 server_base.cc:1034] running on GCE node
I20260929 20:28:12.543853 7286 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:28:12.545368 7286 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:28:12.546597 7286 hybrid_clock.cc:648] HybridClock initialized: now 1790713692546553 us; error 33 us; skew 500 ppm
I20260929 20:28:12.547039 7286 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:28:12 dist-test-slave-zbnc krb5kdc[7273](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713692, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.53.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:28:12.550379 7286 init.cc:377] Logged in from keytab as kudu/127.17.53.126@KRBTEST.COM (short username kudu)
I20260929 20:28:12.552039 7286 webserver.cc:460] Webserver started at http://127.17.53.126:42559/ using document root <none> and password file <none>
I20260929 20:28:12.552811 7286 fs_manager.cc:359] Metadata directory not provided
I20260929 20:28:12.552924 7286 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:28:12.553210 7286 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:28:12.556609 7286 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790713339161492-17621-0/minicluster-data/master-0/data/instance:
uuid: "0b268ef4b3984d39a16791a41c53a1d4"
format_stamp: "Formatted at 2026-09-29 20:28:12 on dist-test-slave-zbnc"
server_key: "3322feaaeab951583958a8bc22f91f67"
server_key_iv: "c97c2912a131af0980459336cf5378da"
server_key_version: "encryptionkey@0"
I20260929 20:28:12.557426 7286 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790713339161492-17621-0/minicluster-data/master-0/wal/instance:
uuid: "0b268ef4b3984d39a16791a41c53a1d4"
format_stamp: "Formatted at 2026-09-29 20:28:12 on dist-test-slave-zbnc"
server_key: "3322feaaeab951583958a8bc22f91f67"
server_key_iv: "c97c2912a131af0980459336cf5378da"
server_key_version: "encryptionkey@0"
I20260929 20:28:12.563094 7286 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260929 20:28:12.567128 7301 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:28:12.568668 7286 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 20:28:12.568825 7286 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790713339161492-17621-0/minicluster-data/master-0/data,/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790713339161492-17621-0/minicluster-data/master-0/wal
uuid: "0b268ef4b3984d39a16791a41c53a1d4"
format_stamp: "Formatted at 2026-09-29 20:28:12 on dist-test-slave-zbnc"
server_key: "3322feaaeab951583958a8bc22f91f67"
server_key_iv: "c97c2912a131af0980459336cf5378da"
server_key_version: "encryptionkey@0"
I20260929 20:28:12.569008 7286 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790713339161492-17621-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790713339161492-17621-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790713339161492-17621-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:28:12.583627 7286 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:28:12.587778 7286 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:28:12.588048 7286 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:28:12.607034 7286 rpc_server.cc:307] RPC server started. Bound to: 127.17.53.126:37885
I20260929 20:28:12.607043 7352 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.53.126:37885 every 8 connection(s)
I20260929 20:28:12.608515 7286 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790713339161492-17621-0/minicluster-data/master-0/data/info.pb
I20260929 20:28:12.612325 17621 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu as pid 7286
I20260929 20:28:12.612532 17621 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790713339161492-17621-0/minicluster-data/master-0/wal/instance
I20260929 20:28:12.612635 7353 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:28:12.613096 17621 external_mini_cluster.cc:1428] Setting key 1908d480c0937b721372829608d3354d
I20260929 20:28:12.627619 7353 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b268ef4b3984d39a16791a41c53a1d4: Bootstrap starting.
Sep 29 20:28:12 dist-test-slave-zbnc krb5kdc[7273](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713692, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.53.126@KRBTEST.COM
I20260929 20:28:12.631466 7353 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0b268ef4b3984d39a16791a41c53a1d4: Neither blocks nor log segments found. Creating new log.
I20260929 20:28:12.632603 7353 log.cc:824] T 00000000000000000000000000000000 P 0b268ef4b3984d39a16791a41c53a1d4: Log is configured to *not* fsync() on all Append() calls
I20260929 20:28:12.635764 7353 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b268ef4b3984d39a16791a41c53a1d4: No bootstrap required, opened a new log
I20260929 20:28:12.637682 7356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:28:12.614976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42747 (local address 127.17.53.126:37885)
0929 20:28:12.615827 (+ 851us) server_negotiation.cc:206] Beginning negotiation
0929 20:28:12.615838 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 20:28:12.615880 (+ 42us) server_negotiation.cc:407] Connection header received
0929 20:28:12.617489 (+ 1609us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:28:12.617510 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:28:12.617919 (+ 409us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:28:12.618245 (+ 326us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:28:12.620196 (+ 1951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:28:12.621836 (+ 1640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:28:12.623794 (+ 1958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:28:12.624751 (+ 957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:28:12.624931 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:28:12.629868 (+ 4937us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:28:12.629879 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:28:12.629899 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:28:12.629946 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:28:12.633494 (+ 3548us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:28:12.634224 (+ 730us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:28:12.634231 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:28:12.634238 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:28:12.634409 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:28:12.634813 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:28:12.634818 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:28:12.634821 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:28:12.635473 (+ 652us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:28:12.635808 (+ 335us) server_negotiation.cc:1035] Waiting for connection context
0929 20:28:12.636298 (+ 490us) server_negotiation.cc:299] Negotiation successful
0929 20:28:12.636709 (+ 411us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":628,"thread_start_us":380,"threads_started":1}
I20260929 20:28:12.641352 17621 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 20:28:12.643898 7353 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0b268ef4b3984d39a16791a41c53a1d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b268ef4b3984d39a16791a41c53a1d4" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 37885 } }
I20260929 20:28:12.644407 7353 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0b268ef4b3984d39a16791a41c53a1d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:28:12.644528 7353 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0b268ef4b3984d39a16791a41c53a1d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b268ef4b3984d39a16791a41c53a1d4, State: Initialized, Role: FOLLOWER
I20260929 20:28:12.645107 7353 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0b268ef4b3984d39a16791a41c53a1d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b268ef4b3984d39a16791a41c53a1d4" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 37885 } }
I20260929 20:28:12.645361 7353 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0b268ef4b3984d39a16791a41c53a1d4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 20:28:12.645524 7353 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0b268ef4b3984d39a16791a41c53a1d4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 20:28:12.645735 7353 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0b268ef4b3984d39a16791a41c53a1d4 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:28:12.649176 7353 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0b268ef4b3984d39a16791a41c53a1d4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b268ef4b3984d39a16791a41c53a1d4" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 37885 } }
I20260929 20:28:12.649794 7353 leader_election.cc:302] T 00000000000000000000000000000000 P 0b268ef4b3984d39a16791a41c53a1d4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0b268ef4b3984d39a16791a41c53a1d4; no voters:
I20260929 20:28:12.650453 7353 leader_election.cc:288] T 00000000000000000000000000000000 P 0b268ef4b3984d39a16791a41c53a1d4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 20:28:12.650568 7363 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0b268ef4b3984d39a16791a41c53a1d4 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:28:12.652886 7353 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0b268ef4b3984d39a16791a41c53a1d4 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:28:12.655164 7363 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0b268ef4b3984d39a16791a41c53a1d4 [term 1 LEADER]: Becoming Leader. State: Replica: 0b268ef4b3984d39a16791a41c53a1d4, State: Running, Role: LEADER
I20260929 20:28:12.659034 7363 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0b268ef4b3984d39a16791a41c53a1d4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b268ef4b3984d39a16791a41c53a1d4" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 37885 } }
I20260929 20:28:12.662535 7364 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b268ef4b3984d39a16791a41c53a1d4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0b268ef4b3984d39a16791a41c53a1d4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b268ef4b3984d39a16791a41c53a1d4" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 37885 } } }
I20260929 20:28:12.663048 7364 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b268ef4b3984d39a16791a41c53a1d4 [sys.catalog]: This master's current role is: LEADER
I20260929 20:28:12.663633 7365 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b268ef4b3984d39a16791a41c53a1d4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0b268ef4b3984d39a16791a41c53a1d4. Latest consensus state: current_term: 1 leader_uuid: "0b268ef4b3984d39a16791a41c53a1d4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b268ef4b3984d39a16791a41c53a1d4" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 37885 } } }
I20260929 20:28:12.664017 7365 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b268ef4b3984d39a16791a41c53a1d4 [sys.catalog]: This master's current role is: LEADER
I20260929 20:28:12.667850 7356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:28:12.645821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46217 (local address 127.17.53.126:37885)
0929 20:28:12.646020 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0929 20:28:12.646029 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 20:28:12.646521 (+ 492us) server_negotiation.cc:407] Connection header received
0929 20:28:12.646698 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:28:12.646702 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:28:12.646762 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:28:12.646936 (+ 174us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:28:12.648734 (+ 1798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:28:12.649463 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:28:12.651465 (+ 2002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:28:12.652120 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:28:12.652222 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:28:12.656075 (+ 3853us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:28:12.656083 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:28:12.656091 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:28:12.656141 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:28:12.662915 (+ 6774us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:28:12.663812 (+ 897us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:28:12.663818 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:28:12.663821 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:28:12.664018 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:28:12.664532 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:28:12.664539 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:28:12.664545 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:28:12.665062 (+ 517us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:28:12.665410 (+ 348us) server_negotiation.cc:1035] Waiting for connection context
0929 20:28:12.667038 (+ 1628us) server_negotiation.cc:299] Negotiation successful
0929 20:28:12.667340 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260929 20:28:12.671373 7375 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:28:12.676801 7375 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:28:12.685452 7375 catalog_manager.cc:1347] Generated new cluster ID: 7ebf144b2e8b43e1941b683681475803
I20260929 20:28:12.685592 7375 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:28:12.700474 7375 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 20:28:12.701750 7375 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:28:12.714993 7375 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0b268ef4b3984d39a16791a41c53a1d4: Generated new TSK 0
I20260929 20:28:12.716151 7375 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 20:28:12.728488 17621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu pid 7286
I20260929 20:28:12.729718 7348 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:28:12.821875 17621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu with pid 7286
2026-09-29T20:28:12Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (2048 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790713339161492-17621-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 20:28:12 dist-test-slave-zbnc krb5kdc[7385](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 20:28:12 dist-test-slave-zbnc krb5kdc[7385](info): set up 2 sockets
Sep 29 20:28:12 dist-test-slave-zbnc krb5kdc[7385](info): commencing operation
krb5kdc: starting...
W20260929 20:28:13.980813 17621 mini_kdc.cc:121] Time spent starting KDC: real 1.123s user 0.000s sys 0.033s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790713339161492-17621-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790713339161492-17621-0/krb5kdc/test-user.keytab.
Sep 29 20:28:14 dist-test-slave-zbnc krb5kdc[7385](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713694, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T20:28:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T20:28:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.17.53.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.53.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.53.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.126.keytab.
Entry for principal kudu/127.17.53.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.126.keytab.
I20260929 20:28:14.242705 17621 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
/tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790713339161492-17621-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790713339161492-17621-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790713339161492-17621-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790713339161492-17621-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.53.126:33435
--webserver_interface=127.17.53.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.126.keytab
--principal=kudu/127.17.53.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.53.84:44179
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.53.126:33435
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790713339161492-17621-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790713339161492-17621-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790713339161492-17621-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:28:14.577143 7398 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:28:14.577440 7398 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:28:14.577632 7398 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:28:14.585212 7398 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:28:14.585381 7398 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:28:14.585472 7398 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:28:14.585570 7398 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:28:14.585672 7398 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:28:14.595985 7398 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.53.84:44179
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790713339161492-17621-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790713339161492-17621-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.53.126:33435
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.126.keytab
--principal=kudu/127.17.53.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.53.126:33435
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790713339161492-17621-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.53.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790713339161492-17621-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:28:14.597260 7398 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:28:14.598538 7398 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:28:14.610651 7406 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:28:14.610881 7404 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:28:14.611023 7403 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:28:14.611395 7398 server_base.cc:1034] running on GCE node
I20260929 20:28:14.612313 7398 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:28:14.613842 7398 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:28:14.615077 7398 hybrid_clock.cc:648] HybridClock initialized: now 1790713694615024 us; error 44 us; skew 500 ppm
I20260929 20:28:14.615469 7398 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:28:14 dist-test-slave-zbnc krb5kdc[7385](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713694, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.53.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:28:14.618614 7398 init.cc:377] Logged in from keytab as kudu/127.17.53.126@KRBTEST.COM (short username kudu)
I20260929 20:28:14.620247 7398 webserver.cc:460] Webserver started at http://127.17.53.126:43577/ using document root <none> and password file <none>
I20260929 20:28:14.620954 7398 fs_manager.cc:359] Metadata directory not provided
I20260929 20:28:14.621102 7398 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:28:14.621419 7398 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:28:14.624472 7398 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790713339161492-17621-0/minicluster-data/master-0/data/instance:
uuid: "b6aa1292b6264bda82b3127b067c3c6b"
format_stamp: "Formatted at 2026-09-29 20:28:14 on dist-test-slave-zbnc"
server_key: "4f8a4793a6ab50184cdb10af2666e9a6"
server_key_iv: "3779546bc4fa70b51cff5a55fb4e9dc5"
server_key_version: "encryptionkey@0"
I20260929 20:28:14.625236 7398 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790713339161492-17621-0/minicluster-data/master-0/wal/instance:
uuid: "b6aa1292b6264bda82b3127b067c3c6b"
format_stamp: "Formatted at 2026-09-29 20:28:14 on dist-test-slave-zbnc"
server_key: "4f8a4793a6ab50184cdb10af2666e9a6"
server_key_iv: "3779546bc4fa70b51cff5a55fb4e9dc5"
server_key_version: "encryptionkey@0"
I20260929 20:28:14.630393 7398 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 20:28:14.634033 7413 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:28:14.635504 7398 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 20:28:14.635692 7398 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790713339161492-17621-0/minicluster-data/master-0/data,/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790713339161492-17621-0/minicluster-data/master-0/wal
uuid: "b6aa1292b6264bda82b3127b067c3c6b"
format_stamp: "Formatted at 2026-09-29 20:28:14 on dist-test-slave-zbnc"
server_key: "4f8a4793a6ab50184cdb10af2666e9a6"
server_key_iv: "3779546bc4fa70b51cff5a55fb4e9dc5"
server_key_version: "encryptionkey@0"
I20260929 20:28:14.635895 7398 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790713339161492-17621-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790713339161492-17621-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790713339161492-17621-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:28:14.674629 7398 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:28:14.679397 7398 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:28:14.679724 7398 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:28:14.698642 7398 rpc_server.cc:307] RPC server started. Bound to: 127.17.53.126:33435
I20260929 20:28:14.698665 7464 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.53.126:33435 every 8 connection(s)
I20260929 20:28:14.700131 7398 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790713339161492-17621-0/minicluster-data/master-0/data/info.pb
I20260929 20:28:14.704016 7465 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:28:14.708292 17621 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu as pid 7398
I20260929 20:28:14.708473 17621 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790713339161492-17621-0/minicluster-data/master-0/wal/instance
I20260929 20:28:14.709084 17621 external_mini_cluster.cc:1428] Setting key 65a06db98c817a3266f13a850c4cc38c
I20260929 20:28:14.715969 7465 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b6aa1292b6264bda82b3127b067c3c6b: Bootstrap starting.
I20260929 20:28:14.719702 7465 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b6aa1292b6264bda82b3127b067c3c6b: Neither blocks nor log segments found. Creating new log.
I20260929 20:28:14.721855 7465 log.cc:824] T 00000000000000000000000000000000 P b6aa1292b6264bda82b3127b067c3c6b: Log is configured to *not* fsync() on all Append() calls
I20260929 20:28:14.725165 7468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:28:14.711122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44431 (local address 127.17.53.126:33435)
0929 20:28:14.711960 (+ 838us) server_negotiation.cc:206] Beginning negotiation
0929 20:28:14.711975 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0929 20:28:14.712017 (+ 42us) server_negotiation.cc:407] Connection header received
0929 20:28:14.714343 (+ 2326us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:28:14.714368 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:28:14.714914 (+ 546us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:28:14.715295 (+ 381us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:28:14.715940 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:28:14.717325 (+ 1385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:28:14.719212 (+ 1887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:28:14.720012 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:28:14.720161 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:28:14.722022 (+ 1861us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:28:14.722037 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:28:14.722063 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:28:14.722918 (+ 855us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:28:14.723029 (+ 111us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:28:14.723065 (+ 36us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:28:14.723192 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0929 20:28:14.723497 (+ 305us) server_negotiation.cc:299] Negotiation successful
0929 20:28:14.723804 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":627,"thread_start_us":390,"threads_started":1}
I20260929 20:28:14.726397 7465 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b6aa1292b6264bda82b3127b067c3c6b: No bootstrap required, opened a new log
I20260929 20:28:14.735360 17621 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 20:28:14.738480 7465 raft_consensus.cc:348] T 00000000000000000000000000000000 P b6aa1292b6264bda82b3127b067c3c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6aa1292b6264bda82b3127b067c3c6b" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 33435 } }
I20260929 20:28:14.739169 7465 raft_consensus.cc:374] T 00000000000000000000000000000000 P b6aa1292b6264bda82b3127b067c3c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:28:14.739315 7465 raft_consensus.cc:729] T 00000000000000000000000000000000 P b6aa1292b6264bda82b3127b067c3c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6aa1292b6264bda82b3127b067c3c6b, State: Initialized, Role: FOLLOWER
I20260929 20:28:14.740206 7465 consensus_queue.cc:260] T 00000000000000000000000000000000 P b6aa1292b6264bda82b3127b067c3c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6aa1292b6264bda82b3127b067c3c6b" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 33435 } }
I20260929 20:28:14.740724 7465 raft_consensus.cc:388] T 00000000000000000000000000000000 P b6aa1292b6264bda82b3127b067c3c6b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 20:28:14.740967 7465 raft_consensus.cc:482] T 00000000000000000000000000000000 P b6aa1292b6264bda82b3127b067c3c6b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 20:28:14.741254 7465 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b6aa1292b6264bda82b3127b067c3c6b [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:28:14.745003 7465 raft_consensus.cc:504] T 00000000000000000000000000000000 P b6aa1292b6264bda82b3127b067c3c6b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6aa1292b6264bda82b3127b067c3c6b" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 33435 } }
I20260929 20:28:14.745712 7465 leader_election.cc:302] T 00000000000000000000000000000000 P b6aa1292b6264bda82b3127b067c3c6b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b6aa1292b6264bda82b3127b067c3c6b; no voters:
I20260929 20:28:14.746447 7465 leader_election.cc:288] T 00000000000000000000000000000000 P b6aa1292b6264bda82b3127b067c3c6b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 20:28:14.746595 7475 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b6aa1292b6264bda82b3127b067c3c6b [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:28:14.749060 7465 sys_catalog.cc:564] T 00000000000000000000000000000000 P b6aa1292b6264bda82b3127b067c3c6b [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:28:14.749956 7475 raft_consensus.cc:686] T 00000000000000000000000000000000 P b6aa1292b6264bda82b3127b067c3c6b [term 1 LEADER]: Becoming Leader. State: Replica: b6aa1292b6264bda82b3127b067c3c6b, State: Running, Role: LEADER
I20260929 20:28:14.750664 7475 consensus_queue.cc:237] T 00000000000000000000000000000000 P b6aa1292b6264bda82b3127b067c3c6b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6aa1292b6264bda82b3127b067c3c6b" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 33435 } }
I20260929 20:28:14.751765 7468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:28:14.740442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38337 (local address 127.17.53.126:33435)
0929 20:28:14.740651 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0929 20:28:14.740657 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:28:14.743521 (+ 2864us) server_negotiation.cc:407] Connection header received
0929 20:28:14.743823 (+ 302us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:28:14.743828 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:28:14.743896 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:28:14.744025 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:28:14.744642 (+ 617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:28:14.745429 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:28:14.747339 (+ 1910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:28:14.748345 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:28:14.748480 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:28:14.750440 (+ 1960us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:28:14.750446 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:28:14.750451 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:28:14.750476 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:28:14.750536 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:28:14.750559 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:28:14.750658 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0929 20:28:14.751209 (+ 551us) server_negotiation.cc:299] Negotiation successful
0929 20:28:14.751314 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260929 20:28:14.755707 7477 sys_catalog.cc:455] T 00000000000000000000000000000000 P b6aa1292b6264bda82b3127b067c3c6b [sys.catalog]: SysCatalogTable state changed. Reason: New leader b6aa1292b6264bda82b3127b067c3c6b. Latest consensus state: current_term: 1 leader_uuid: "b6aa1292b6264bda82b3127b067c3c6b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6aa1292b6264bda82b3127b067c3c6b" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 33435 } } }
I20260929 20:28:14.755695 7476 sys_catalog.cc:455] T 00000000000000000000000000000000 P b6aa1292b6264bda82b3127b067c3c6b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b6aa1292b6264bda82b3127b067c3c6b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6aa1292b6264bda82b3127b067c3c6b" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 33435 } } }
I20260929 20:28:14.756332 7477 sys_catalog.cc:458] T 00000000000000000000000000000000 P b6aa1292b6264bda82b3127b067c3c6b [sys.catalog]: This master's current role is: LEADER
I20260929 20:28:14.756388 7476 sys_catalog.cc:458] T 00000000000000000000000000000000 P b6aa1292b6264bda82b3127b067c3c6b [sys.catalog]: This master's current role is: LEADER
I20260929 20:28:14.758306 7482 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:28:14.766204 7482 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:28:14.779790 7482 catalog_manager.cc:1347] Generated new cluster ID: 109f87a55e074abfbbebed76a03c9d43
I20260929 20:28:14.779917 7482 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:28:14.812889 7482 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 20:28:14.814256 7482 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:28:14.826483 7482 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b6aa1292b6264bda82b3127b067c3c6b: Generated new TSK 0
I20260929 20:28:14.827495 7482 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 20:28:14.841552 17621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu pid 7398
W20260929 20:28:14.842684 7460 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:44431
W20260929 20:28:14.843472 17621 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu pid 7398: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 20:28:14.843626 17621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu with pid 7398
2026-09-29T20:28:14Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (2021 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790713339161492-17621-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 20:28:14 dist-test-slave-zbnc krb5kdc[7496](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 20:28:14 dist-test-slave-zbnc krb5kdc[7496](info): set up 2 sockets
Sep 29 20:28:14 dist-test-slave-zbnc krb5kdc[7496](info): commencing operation
krb5kdc: starting...
W20260929 20:28:15.968804 17621 mini_kdc.cc:121] Time spent starting KDC: real 1.089s user 0.001s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790713339161492-17621-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790713339161492-17621-0/krb5kdc/test-user.keytab.
Sep 29 20:28:16 dist-test-slave-zbnc krb5kdc[7496](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713696, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T20:28:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T20:28:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.17.53.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.53.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.53.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.126.keytab.
Entry for principal kudu/127.17.53.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.126.keytab.
I20260929 20:28:16.235937 17621 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
/tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790713339161492-17621-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790713339161492-17621-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790713339161492-17621-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790713339161492-17621-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.53.126:45723
--webserver_interface=127.17.53.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.126.keytab
--principal=kudu/127.17.53.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.53.84:45241
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.53.126:45723
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790713339161492-17621-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790713339161492-17621-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790713339161492-17621-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:28:16.565876 7509 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:28:16.566154 7509 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:28:16.566332 7509 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:28:16.573503 7509 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:28:16.573616 7509 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:28:16.573705 7509 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:28:16.573792 7509 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:28:16.573876 7509 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:28:16.583639 7509 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.53.84:45241
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790713339161492-17621-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790713339161492-17621-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.53.126:45723
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.126.keytab
--principal=kudu/127.17.53.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.53.126:45723
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790713339161492-17621-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.53.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790713339161492-17621-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:28:16.585088 7509 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:28:16.586499 7509 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:28:16.598587 7514 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:28:16.598585 7515 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:28:16.598829 7517 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:28:16.599681 7509 server_base.cc:1034] running on GCE node
I20260929 20:28:16.600446 7509 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:28:16.601895 7509 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:28:16.603130 7509 hybrid_clock.cc:648] HybridClock initialized: now 1790713696603061 us; error 65 us; skew 500 ppm
I20260929 20:28:16.603485 7509 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:28:16 dist-test-slave-zbnc krb5kdc[7496](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713696, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.53.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:28:16.610098 7509 init.cc:377] Logged in from keytab as kudu/127.17.53.126@KRBTEST.COM (short username kudu)
I20260929 20:28:16.611842 7509 webserver.cc:460] Webserver started at http://127.17.53.126:39373/ using document root <none> and password file <none>
I20260929 20:28:16.612581 7509 fs_manager.cc:359] Metadata directory not provided
I20260929 20:28:16.612739 7509 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:28:16.613056 7509 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:28:16.616174 7509 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790713339161492-17621-0/minicluster-data/master-0/data/instance:
uuid: "d3138f7fdbe74b6fb141ff3d1b794a7f"
format_stamp: "Formatted at 2026-09-29 20:28:16 on dist-test-slave-zbnc"
server_key: "818f1a779e03ac76020fe722a61b8830"
server_key_iv: "5fc03ec832c81e10daa615ae8f0de547"
server_key_version: "encryptionkey@0"
I20260929 20:28:16.616961 7509 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790713339161492-17621-0/minicluster-data/master-0/wal/instance:
uuid: "d3138f7fdbe74b6fb141ff3d1b794a7f"
format_stamp: "Formatted at 2026-09-29 20:28:16 on dist-test-slave-zbnc"
server_key: "818f1a779e03ac76020fe722a61b8830"
server_key_iv: "5fc03ec832c81e10daa615ae8f0de547"
server_key_version: "encryptionkey@0"
I20260929 20:28:16.622594 7509 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 20:28:16.626556 7524 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:28:16.628180 7509 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 20:28:16.628376 7509 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790713339161492-17621-0/minicluster-data/master-0/data,/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790713339161492-17621-0/minicluster-data/master-0/wal
uuid: "d3138f7fdbe74b6fb141ff3d1b794a7f"
format_stamp: "Formatted at 2026-09-29 20:28:16 on dist-test-slave-zbnc"
server_key: "818f1a779e03ac76020fe722a61b8830"
server_key_iv: "5fc03ec832c81e10daa615ae8f0de547"
server_key_version: "encryptionkey@0"
I20260929 20:28:16.628579 7509 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790713339161492-17621-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790713339161492-17621-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790713339161492-17621-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:28:16.634502 7509 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:28:16.639025 7509 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:28:16.639351 7509 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:28:16.661087 7509 rpc_server.cc:307] RPC server started. Bound to: 127.17.53.126:45723
I20260929 20:28:16.661087 7575 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.53.126:45723 every 8 connection(s)
I20260929 20:28:16.662755 7509 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790713339161492-17621-0/minicluster-data/master-0/data/info.pb
I20260929 20:28:16.666541 7576 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:28:16.672015 17621 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu as pid 7509
I20260929 20:28:16.672200 17621 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790713339161492-17621-0/minicluster-data/master-0/wal/instance
I20260929 20:28:16.672715 17621 external_mini_cluster.cc:1428] Setting key aba5305db429865c2825cd088c31a21a
I20260929 20:28:16.678181 7576 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d3138f7fdbe74b6fb141ff3d1b794a7f: Bootstrap starting.
I20260929 20:28:16.680779 7579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:28:16.674653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42247 (local address 127.17.53.126:45723)
0929 20:28:16.675458 (+ 805us) server_negotiation.cc:206] Beginning negotiation
0929 20:28:16.675472 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0929 20:28:16.675514 (+ 42us) server_negotiation.cc:407] Connection header received
0929 20:28:16.677251 (+ 1737us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:28:16.677277 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:28:16.677635 (+ 358us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:28:16.678046 (+ 411us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:28:16.679185 (+ 1139us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:28:16.679191 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:28:16.679205 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:28:16.679219 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:28:16.679278 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:28:16.679297 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:28:16.679555 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0929 20:28:16.679692 (+ 137us) server_negotiation.cc:299] Negotiation successful
0929 20:28:16.679878 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":581,"thread_start_us":350,"threads_started":1}
I20260929 20:28:16.681962 7576 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d3138f7fdbe74b6fb141ff3d1b794a7f: Neither blocks nor log segments found. Creating new log.
I20260929 20:28:16.683149 7576 log.cc:824] T 00000000000000000000000000000000 P d3138f7fdbe74b6fb141ff3d1b794a7f: Log is configured to *not* fsync() on all Append() calls
I20260929 20:28:16.684667 17621 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 20:28:16.686095 7576 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d3138f7fdbe74b6fb141ff3d1b794a7f: No bootstrap required, opened a new log
I20260929 20:28:16.691671 7579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:28:16.689163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59517 (local address 127.17.53.126:45723)
0929 20:28:16.689349 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0929 20:28:16.689356 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:28:16.689588 (+ 232us) server_negotiation.cc:407] Connection header received
0929 20:28:16.689868 (+ 280us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:28:16.689872 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:28:16.689921 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:28:16.690006 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:28:16.690979 (+ 973us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:28:16.690983 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:28:16.690986 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0929 20:28:16.691000 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:28:16.691036 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0929 20:28:16.691055 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:28:16.691151 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0929 20:28:16.691288 (+ 137us) server_negotiation.cc:299] Negotiation successful
0929 20:28:16.691336 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260929 20:28:16.695353 7576 raft_consensus.cc:348] T 00000000000000000000000000000000 P d3138f7fdbe74b6fb141ff3d1b794a7f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3138f7fdbe74b6fb141ff3d1b794a7f" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 45723 } }
I20260929 20:28:16.695710 7576 raft_consensus.cc:374] T 00000000000000000000000000000000 P d3138f7fdbe74b6fb141ff3d1b794a7f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 20:28:16.695775 7576 raft_consensus.cc:729] T 00000000000000000000000000000000 P d3138f7fdbe74b6fb141ff3d1b794a7f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3138f7fdbe74b6fb141ff3d1b794a7f, State: Initialized, Role: FOLLOWER
I20260929 20:28:16.696339 7576 consensus_queue.cc:260] T 00000000000000000000000000000000 P d3138f7fdbe74b6fb141ff3d1b794a7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3138f7fdbe74b6fb141ff3d1b794a7f" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 45723 } }
I20260929 20:28:16.696614 7576 raft_consensus.cc:388] T 00000000000000000000000000000000 P d3138f7fdbe74b6fb141ff3d1b794a7f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 20:28:16.696780 7576 raft_consensus.cc:482] T 00000000000000000000000000000000 P d3138f7fdbe74b6fb141ff3d1b794a7f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 20:28:16.696971 7576 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d3138f7fdbe74b6fb141ff3d1b794a7f [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:28:16.699301 7576 raft_consensus.cc:504] T 00000000000000000000000000000000 P d3138f7fdbe74b6fb141ff3d1b794a7f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3138f7fdbe74b6fb141ff3d1b794a7f" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 45723 } }
I20260929 20:28:16.699820 7576 leader_election.cc:302] T 00000000000000000000000000000000 P d3138f7fdbe74b6fb141ff3d1b794a7f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d3138f7fdbe74b6fb141ff3d1b794a7f; no voters:
I20260929 20:28:16.700423 7576 leader_election.cc:288] T 00000000000000000000000000000000 P d3138f7fdbe74b6fb141ff3d1b794a7f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 20:28:16.700589 7586 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d3138f7fdbe74b6fb141ff3d1b794a7f [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:28:16.701246 7586 raft_consensus.cc:686] T 00000000000000000000000000000000 P d3138f7fdbe74b6fb141ff3d1b794a7f [term 1 LEADER]: Becoming Leader. State: Replica: d3138f7fdbe74b6fb141ff3d1b794a7f, State: Running, Role: LEADER
I20260929 20:28:16.701896 7586 consensus_queue.cc:237] T 00000000000000000000000000000000 P d3138f7fdbe74b6fb141ff3d1b794a7f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3138f7fdbe74b6fb141ff3d1b794a7f" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 45723 } }
I20260929 20:28:16.702795 7576 sys_catalog.cc:564] T 00000000000000000000000000000000 P d3138f7fdbe74b6fb141ff3d1b794a7f [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:28:16.705143 7588 sys_catalog.cc:455] T 00000000000000000000000000000000 P d3138f7fdbe74b6fb141ff3d1b794a7f [sys.catalog]: SysCatalogTable state changed. Reason: New leader d3138f7fdbe74b6fb141ff3d1b794a7f. Latest consensus state: current_term: 1 leader_uuid: "d3138f7fdbe74b6fb141ff3d1b794a7f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3138f7fdbe74b6fb141ff3d1b794a7f" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 45723 } } }
I20260929 20:28:16.705135 7587 sys_catalog.cc:455] T 00000000000000000000000000000000 P d3138f7fdbe74b6fb141ff3d1b794a7f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d3138f7fdbe74b6fb141ff3d1b794a7f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d3138f7fdbe74b6fb141ff3d1b794a7f" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 45723 } } }
I20260929 20:28:16.705605 7588 sys_catalog.cc:458] T 00000000000000000000000000000000 P d3138f7fdbe74b6fb141ff3d1b794a7f [sys.catalog]: This master's current role is: LEADER
I20260929 20:28:16.705641 7587 sys_catalog.cc:458] T 00000000000000000000000000000000 P d3138f7fdbe74b6fb141ff3d1b794a7f [sys.catalog]: This master's current role is: LEADER
I20260929 20:28:16.707922 7594 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:28:16.713773 7594 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:28:16.723357 7594 catalog_manager.cc:1347] Generated new cluster ID: 9a24554acf5647a49480a1b0554446ea
I20260929 20:28:16.723498 7594 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:28:16.736367 7594 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 20:28:16.736881 7594 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:28:16.751016 7594 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d3138f7fdbe74b6fb141ff3d1b794a7f: Generated new TSK 0
I20260929 20:28:16.751904 7594 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 20:28:16.757673 17621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu pid 7509
W20260929 20:28:16.758924 7571 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:42247
W20260929 20:28:16.759685 17621 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu pid 7509: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260929 20:28:16.759855 17621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu with pid 7509
2026-09-29T20:28:16Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1915 ms)
[----------] 18 tests from AuthTokenIssuingTest (35493 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790713339161492-17621-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 20:28:16 dist-test-slave-zbnc krb5kdc[7607](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 20:28:16 dist-test-slave-zbnc krb5kdc[7607](info): set up 2 sockets
Sep 29 20:28:16 dist-test-slave-zbnc krb5kdc[7607](info): commencing operation
krb5kdc: starting...
W20260929 20:28:17.869001 17621 mini_kdc.cc:121] Time spent starting KDC: real 1.073s user 0.000s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790713339161492-17621-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790713339161492-17621-0/krb5kdc/test-user.keytab.
Sep 29 20:28:18 dist-test-slave-zbnc krb5kdc[7607](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713698, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T20:28:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T20:28:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.17.53.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.53.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.53.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.126.keytab.
Entry for principal kudu/127.17.53.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.126.keytab.
I20260929 20:28:18.135095 17621 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
/tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790713339161492-17621-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790713339161492-17621-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790713339161492-17621-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790713339161492-17621-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.53.126:33377
--webserver_interface=127.17.53.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.126.keytab
--principal=kudu/127.17.53.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.53.84:39951
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.53.126:33377,127.17.53.125:43633,127.17.53.124:41889
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790713339161492-17621-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790713339161492-17621-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790713339161492-17621-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:28:18.468101 7620 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 20:28:18.468479 7620 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:28:18.468604 7620 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:28:18.468755 7620 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:28:18.476063 7620 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:28:18.476155 7620 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:28:18.476256 7620 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:28:18.476336 7620 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:28:18.476425 7620 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:28:18.486469 7620 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.53.84:39951
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790713339161492-17621-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790713339161492-17621-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.53.126:33377,127.17.53.125:43633,127.17.53.124:41889
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.126.keytab
--principal=kudu/127.17.53.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.53.126:33377
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790713339161492-17621-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.53.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790713339161492-17621-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:28:18.487879 7620 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:28:18.489283 7620 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:28:18.501507 7628 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:28:18.501456 7626 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:28:18.501685 7625 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:28:18.502489 7620 server_base.cc:1034] running on GCE node
I20260929 20:28:18.503352 7620 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:28:18.504702 7620 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:28:18.505925 7620 hybrid_clock.cc:648] HybridClock initialized: now 1790713698505867 us; error 29 us; skew 500 ppm
I20260929 20:28:18.506300 7620 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:28:18 dist-test-slave-zbnc krb5kdc[7607](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713698, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.53.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:28:18.509591 7620 init.cc:377] Logged in from keytab as kudu/127.17.53.126@KRBTEST.COM (short username kudu)
I20260929 20:28:18.511080 7620 webserver.cc:460] Webserver started at http://127.17.53.126:46353/ using document root <none> and password file <none>
I20260929 20:28:18.511750 7620 fs_manager.cc:359] Metadata directory not provided
I20260929 20:28:18.511842 7620 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:28:18.512108 7620 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:28:18.515205 7620 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790713339161492-17621-0/minicluster-data/master-0/data/instance:
uuid: "3f9b2f0ee39b4efa93370234003a35d0"
format_stamp: "Formatted at 2026-09-29 20:28:18 on dist-test-slave-zbnc"
server_key: "382fbe0bca3df36af92c2019708fcc5a"
server_key_iv: "88b4f43edcd76e951e52566f26e0bd1e"
server_key_version: "encryptionkey@0"
I20260929 20:28:18.515973 7620 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790713339161492-17621-0/minicluster-data/master-0/wal/instance:
uuid: "3f9b2f0ee39b4efa93370234003a35d0"
format_stamp: "Formatted at 2026-09-29 20:28:18 on dist-test-slave-zbnc"
server_key: "382fbe0bca3df36af92c2019708fcc5a"
server_key_iv: "88b4f43edcd76e951e52566f26e0bd1e"
server_key_version: "encryptionkey@0"
I20260929 20:28:18.521449 7620 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 20:28:18.525251 7635 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:28:18.526659 7620 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 20:28:18.526940 7620 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790713339161492-17621-0/minicluster-data/master-0/data,/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790713339161492-17621-0/minicluster-data/master-0/wal
uuid: "3f9b2f0ee39b4efa93370234003a35d0"
format_stamp: "Formatted at 2026-09-29 20:28:18 on dist-test-slave-zbnc"
server_key: "382fbe0bca3df36af92c2019708fcc5a"
server_key_iv: "88b4f43edcd76e951e52566f26e0bd1e"
server_key_version: "encryptionkey@0"
I20260929 20:28:18.527184 7620 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790713339161492-17621-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790713339161492-17621-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790713339161492-17621-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:28:18.546463 7620 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:28:18.551144 7620 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:28:18.551438 7620 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:28:18.570793 7620 rpc_server.cc:307] RPC server started. Bound to: 127.17.53.126:33377
I20260929 20:28:18.570935 7686 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.53.126:33377 every 8 connection(s)
I20260929 20:28:18.572296 7620 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790713339161492-17621-0/minicluster-data/master-0/data/info.pb
I20260929 20:28:18.576109 7687 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:28:18.581233 17621 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu as pid 7620
I20260929 20:28:18.581451 17621 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790713339161492-17621-0/minicluster-data/master-0/wal/instance
I20260929 20:28:18.582261 17621 external_mini_cluster.cc:1428] Setting key 12059421e017d940d3060a335aa5e670
I20260929 20:28:18.583667 7687 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.53.126" port: 33377 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 20:28:18 dist-test-slave-zbnc krb5kdc[7607](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713698, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.53.126@KRBTEST.COM for kudu/127.17.53.126@KRBTEST.COM
I20260929 20:28:18.608439 7689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:28:18.587840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:33377 (local address 127.0.0.1:45090)
0929 20:28:18.588814 (+ 974us) negotiation.cc:107] Waiting for socket to connect
0929 20:28:18.588875 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0929 20:28:18.591793 (+ 2918us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:28:18.592708 (+ 915us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:28:18.592715 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:28:18.593457 (+ 742us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:28:18.593802 (+ 345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:28:18.593821 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:28:18.595453 (+ 1632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:28:18.595459 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:28:18.596550 (+ 1091us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:28:18.596561 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:28:18.597494 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:28:18.597501 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:28:18.597686 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:28:18.598350 (+ 664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:28:18.598371 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:28:18.600901 (+ 2530us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:28:18.604226 (+ 3325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:28:18.604239 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:28:18.604253 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:28:18.604775 (+ 522us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:28:18.605262 (+ 487us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:28:18.605265 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:28:18.605267 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:28:18.605403 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:28:18.606447 (+ 1044us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:28:18.606454 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:28:18.606666 (+ 212us) client_negotiation.cc:769] Sending connection context
0929 20:28:18.606982 (+ 316us) client_negotiation.cc:240] Negotiation successful
0929 20:28:18.607397 (+ 415us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":651,"thread_start_us":401,"threads_started":1}
I20260929 20:28:18.608466 7690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:28:18.587840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45090 (local address 127.17.53.126:33377)
0929 20:28:18.589834 (+ 1994us) server_negotiation.cc:206] Beginning negotiation
0929 20:28:18.589852 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0929 20:28:18.589906 (+ 54us) server_negotiation.cc:407] Connection header received
0929 20:28:18.592203 (+ 2297us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:28:18.592209 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:28:18.592471 (+ 262us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:28:18.592700 (+ 229us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:28:18.594022 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:28:18.595304 (+ 1282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:28:18.596699 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:28:18.597252 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:28:18.597457 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:28:18.601123 (+ 3666us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:28:18.601131 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:28:18.601144 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:28:18.601175 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:28:18.604019 (+ 2844us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:28:18.604912 (+ 893us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:28:18.604923 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:28:18.604928 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:28:18.605141 (+ 213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:28:18.605514 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:28:18.605526 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:28:18.605533 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:28:18.606295 (+ 762us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:28:18.606515 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0929 20:28:18.607143 (+ 628us) server_negotiation.cc:299] Negotiation successful
0929 20:28:18.607599 (+ 456us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":127,"server-negotiator.queue_time_us":1791,"spinlock_wait_cycles":54016,"thread_start_us":1553,"threads_started":1}
WARNING: no policy specified for kudu/127.17.53.125@KRBTEST.COM; defaulting to no policy
I20260929 20:28:18.611395 7687 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.53.125" port: 43633 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.53.125@KRBTEST.COM" created.
I20260929 20:28:18.612517 7689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:28:18.612108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.125:43633 (local address 127.0.0.1:57202)
0929 20:28:18.612296 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0929 20:28:18.612407 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.53.125:43633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260929 20:28:18.612748 7638 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.53.125:43633: connect: Connection refused (error 111)
I20260929 20:28:18.614305 7689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:28:18.613952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.125:43633 (local address 127.0.0.1:57208)
0929 20:28:18.614124 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0929 20:28:18.614188 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.53.125:43633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260929 20:28:18.614526 7687 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.17.53.125:43633: Network error: Client connection negotiation failed: client connection to 127.17.53.125:43633: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.53.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.125.keytab.
Entry for principal kudu/127.17.53.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.125.keytab.
I20260929 20:28:18.642637 17621 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
/tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790713339161492-17621-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790713339161492-17621-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790713339161492-17621-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790713339161492-17621-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.53.125:43633
--webserver_interface=127.17.53.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.125.keytab
--principal=kudu/127.17.53.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.53.84:39951
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.53.126:33377,127.17.53.125:43633,127.17.53.124:41889
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790713339161492-17621-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790713339161492-17621-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790713339161492-17621-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 20:28:18.663892 7687 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.53.125" port: 43633 } attempt: 1
I20260929 20:28:18.665109 7689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:28:18.664674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.125:43633 (local address 127.0.0.1:57220)
0929 20:28:18.664866 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0929 20:28:18.664971 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.53.125:43633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260929 20:28:18.666227 7689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:28:18.665865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.125:43633 (local address 127.0.0.1:57232)
0929 20:28:18.666026 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0929 20:28:18.666091 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.53.125:43633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260929 20:28:18.666409 7687 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.17.53.125:43633: Network error: Client connection negotiation failed: client connection to 127.17.53.125:43633: connect: Connection refused (error 111)
I20260929 20:28:18.734642 7687 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.53.125" port: 43633 } attempt: 2
I20260929 20:28:18.735847 7689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:28:18.735355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.125:43633 (local address 127.0.0.1:57244)
0929 20:28:18.735609 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0929 20:28:18.735699 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.53.125:43633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260929 20:28:18.736878 7689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:28:18.736508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.125:43633 (local address 127.0.0.1:57258)
0929 20:28:18.736670 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0929 20:28:18.736755 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.53.125:43633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260929 20:28:18.737064 7687 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.17.53.125:43633: Network error: Client connection negotiation failed: client connection to 127.17.53.125:43633: connect: Connection refused (error 111)
I20260929 20:28:18.828313 7687 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.53.125" port: 43633 } attempt: 3
I20260929 20:28:18.829586 7689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:28:18.829143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.125:43633 (local address 127.0.0.1:57260)
0929 20:28:18.829363 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0929 20:28:18.829456 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.53.125:43633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260929 20:28:18.830751 7689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:28:18.830393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.125:43633 (local address 127.0.0.1:57272)
0929 20:28:18.830540 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0929 20:28:18.830620 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.53.125:43633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260929 20:28:18.830960 7687 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.17.53.125:43633: Network error: Client connection negotiation failed: client connection to 127.17.53.125:43633: connect: Connection refused (error 111)
W20260929 20:28:18.972635 7694 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 20:28:18.973013 7694 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:28:18.973142 7694 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:28:18.973278 7694 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260929 20:28:18.974246 7687 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.53.125" port: 43633 } attempt: 4
I20260929 20:28:18.975584 7689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:28:18.974988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.125:43633 (local address 127.0.0.1:57288)
0929 20:28:18.975274 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0929 20:28:18.975426 (+ 152us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.53.125:43633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163}
I20260929 20:28:18.976606 7689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:28:18.976252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.125:43633 (local address 127.0.0.1:57294)
0929 20:28:18.976401 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0929 20:28:18.976487 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.53.125:43633: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260929 20:28:18.976814 7687 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.17.53.125:43633: Network error: Client connection negotiation failed: client connection to 127.17.53.125:43633: connect: Connection refused (error 111)
W20260929 20:28:18.981206 7694 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:28:18.981283 7694 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:28:18.981401 7694 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:28:18.981459 7694 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:28:18.981544 7694 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:28:18.991554 7694 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.53.84:39951
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790713339161492-17621-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790713339161492-17621-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.17.53.126:33377,127.17.53.125:43633,127.17.53.124:41889
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.125.keytab
--principal=kudu/127.17.53.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.53.125:43633
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790713339161492-17621-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.17.53.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790713339161492-17621-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:28:18.992892 7694 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:28:18.994200 7694 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:28:19.010890 7699 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:28:19.010947 7702 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:28:19.010917 7700 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:28:19.012017 7694 server_base.cc:1034] running on GCE node
I20260929 20:28:19.012828 7694 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:28:19.014278 7694 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:28:19.015534 7694 hybrid_clock.cc:648] HybridClock initialized: now 1790713699015470 us; error 42 us; skew 500 ppm
I20260929 20:28:19.015878 7694 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:28:19 dist-test-slave-zbnc krb5kdc[7607](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713699, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.53.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:28:19.019182 7694 init.cc:377] Logged in from keytab as kudu/127.17.53.125@KRBTEST.COM (short username kudu)
I20260929 20:28:19.020780 7694 webserver.cc:460] Webserver started at http://127.17.53.125:32965/ using document root <none> and password file <none>
I20260929 20:28:19.021420 7694 fs_manager.cc:359] Metadata directory not provided
I20260929 20:28:19.021507 7694 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:28:19.021782 7694 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:28:19.024762 7694 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790713339161492-17621-0/minicluster-data/master-1/data/instance:
uuid: "200282175f6d4993a9937351cf946aeb"
format_stamp: "Formatted at 2026-09-29 20:28:19 on dist-test-slave-zbnc"
server_key: "19208ac55c6ca62ceeace01baa19c1d9"
server_key_iv: "475d62ac48f1acf62a9f64d0d9100d55"
server_key_version: "encryptionkey@0"
I20260929 20:28:19.025486 7694 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790713339161492-17621-0/minicluster-data/master-1/wal/instance:
uuid: "200282175f6d4993a9937351cf946aeb"
format_stamp: "Formatted at 2026-09-29 20:28:19 on dist-test-slave-zbnc"
server_key: "19208ac55c6ca62ceeace01baa19c1d9"
server_key_iv: "475d62ac48f1acf62a9f64d0d9100d55"
server_key_version: "encryptionkey@0"
I20260929 20:28:19.030573 7694 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260929 20:28:19.034261 7709 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:28:19.035660 7694 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 20:28:19.035893 7694 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790713339161492-17621-0/minicluster-data/master-1/data,/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790713339161492-17621-0/minicluster-data/master-1/wal
uuid: "200282175f6d4993a9937351cf946aeb"
format_stamp: "Formatted at 2026-09-29 20:28:19 on dist-test-slave-zbnc"
server_key: "19208ac55c6ca62ceeace01baa19c1d9"
server_key_iv: "475d62ac48f1acf62a9f64d0d9100d55"
server_key_version: "encryptionkey@0"
I20260929 20:28:19.036106 7694 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790713339161492-17621-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790713339161492-17621-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790713339161492-17621-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:28:19.049841 7694 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:28:19.055369 7694 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:28:19.055663 7694 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:28:19.074970 7694 rpc_server.cc:307] RPC server started. Bound to: 127.17.53.125:43633
I20260929 20:28:19.074997 7760 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.53.125:43633 every 8 connection(s)
I20260929 20:28:19.076339 7694 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790713339161492-17621-0/minicluster-data/master-1/data/info.pb
I20260929 20:28:19.077574 17621 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu as pid 7694
I20260929 20:28:19.077802 17621 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790713339161492-17621-0/minicluster-data/master-1/wal/instance
I20260929 20:28:19.078497 17621 external_mini_cluster.cc:1428] Setting key 330aa0ef76468c06c486ca318033ebf3
I20260929 20:28:19.080349 7761 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:28:19.086763 7761 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.53.126" port: 33377 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 20:28:19 dist-test-slave-zbnc krb5kdc[7607](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713699, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.53.125@KRBTEST.COM for kudu/127.17.53.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.17.53.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.53.124@KRBTEST.COM" created.
I20260929 20:28:19.108541 7690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:28:19.089914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45100 (local address 127.17.53.126:33377)
0929 20:28:19.090137 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0929 20:28:19.090144 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:28:19.091267 (+ 1123us) server_negotiation.cc:407] Connection header received
0929 20:28:19.093150 (+ 1883us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:28:19.093159 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:28:19.093255 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:28:19.093382 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:28:19.094929 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:28:19.096056 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:28:19.097653 (+ 1597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:28:19.098451 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:28:19.098600 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:28:19.102001 (+ 3401us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:28:19.102008 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:28:19.102015 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:28:19.102065 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:28:19.105224 (+ 3159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:28:19.106010 (+ 786us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:28:19.106014 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:28:19.106016 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:28:19.106137 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:28:19.106574 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:28:19.106578 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:28:19.106581 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:28:19.107048 (+ 467us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:28:19.107171 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0929 20:28:19.108103 (+ 932us) server_negotiation.cc:299] Negotiation successful
0929 20:28:19.108277 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260929 20:28:19.110188 7764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:28:19.090275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:33377 (local address 127.0.0.1:45100)
0929 20:28:19.091091 (+ 816us) negotiation.cc:107] Waiting for socket to connect
0929 20:28:19.091143 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0929 20:28:19.092873 (+ 1730us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:28:19.093564 (+ 691us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:28:19.093572 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:28:19.094133 (+ 561us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:28:19.094677 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:28:19.094703 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:28:19.096233 (+ 1530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:28:19.096238 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:28:19.097496 (+ 1258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:28:19.097508 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:28:19.098659 (+ 1151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:28:19.098663 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:28:19.098905 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:28:19.099612 (+ 707us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:28:19.099634 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:28:19.101754 (+ 2120us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:28:19.105352 (+ 3598us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:28:19.105359 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:28:19.105374 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:28:19.105864 (+ 490us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:28:19.106234 (+ 370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:28:19.106237 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:28:19.106239 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:28:19.106419 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:28:19.107188 (+ 769us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:28:19.107196 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:28:19.107685 (+ 489us) client_negotiation.cc:769] Sending connection context
0929 20:28:19.108030 (+ 345us) client_negotiation.cc:240] Negotiation successful
0929 20:28:19.108626 (+ 596us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":538,"thread_start_us":352,"threads_started":1}
I20260929 20:28:19.112053 7761 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.53.125" port: 43633 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 20:28:19 dist-test-slave-zbnc krb5kdc[7607](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713699, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.53.125@KRBTEST.COM for kudu/127.17.53.125@KRBTEST.COM
I20260929 20:28:19.129046 7764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:28:19.112942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.125:43633 (local address 127.0.0.1:57310)
0929 20:28:19.113157 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0929 20:28:19.113178 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 20:28:19.113300 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:28:19.114347 (+ 1047us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:28:19.114354 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:28:19.114891 (+ 537us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:28:19.115111 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:28:19.115126 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:28:19.116509 (+ 1383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:28:19.116515 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:28:19.117886 (+ 1371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:28:19.117898 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:28:19.119272 (+ 1374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:28:19.119282 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:28:19.119574 (+ 292us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:28:19.120232 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:28:19.120249 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:28:19.122270 (+ 2021us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:28:19.126644 (+ 4374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:28:19.126649 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:28:19.126653 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:28:19.127081 (+ 428us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:28:19.127479 (+ 398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:28:19.127483 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:28:19.127485 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:28:19.127598 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:28:19.128340 (+ 742us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:28:19.128344 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:28:19.128471 (+ 127us) client_negotiation.cc:769] Sending connection context
0929 20:28:19.128589 (+ 118us) client_negotiation.cc:240] Negotiation successful
0929 20:28:19.128790 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55}
I20260929 20:28:19.129581 7765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:28:19.113172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57310 (local address 127.17.53.125:43633)
0929 20:28:19.113911 (+ 739us) server_negotiation.cc:206] Beginning negotiation
0929 20:28:19.113916 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:28:19.113941 (+ 25us) server_negotiation.cc:407] Connection header received
0929 20:28:19.113984 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:28:19.113990 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:28:19.114164 (+ 174us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:28:19.114294 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:28:19.115301 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:28:19.116357 (+ 1056us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:28:19.118034 (+ 1677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:28:19.119047 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:28:19.119185 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:28:19.122499 (+ 3314us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:28:19.122516 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:28:19.122538 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:28:19.122579 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:28:19.126501 (+ 3922us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:28:19.127212 (+ 711us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:28:19.127224 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:28:19.127230 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:28:19.127372 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:28:19.127717 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:28:19.127724 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:28:19.127730 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:28:19.128216 (+ 486us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:28:19.128423 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0929 20:28:19.128778 (+ 355us) server_negotiation.cc:299] Negotiation successful
0929 20:28:19.129071 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":584,"thread_start_us":401,"threads_started":1}
I20260929 20:28:19.132236 7761 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.53.124" port: 41889 } has no permanent_uuid. Determining permanent_uuid...
I20260929 20:28:19.133538 7764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:28:19.133119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.124:41889 (local address 127.0.0.1:60748)
0929 20:28:19.133285 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0929 20:28:19.133388 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.53.124:41889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260929 20:28:19.133824 7713 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.53.124:41889: connect: Connection refused (error 111)
I20260929 20:28:19.135478 7764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:28:19.135129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.124:41889 (local address 127.0.0.1:60752)
0929 20:28:19.135308 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0929 20:28:19.135381 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.53.124:41889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260929 20:28:19.135723 7761 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.17.53.124:41889: Network error: Client connection negotiation failed: client connection to 127.17.53.124:41889: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.53.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.124.keytab.
Entry for principal kudu/127.17.53.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.124.keytab.
I20260929 20:28:19.146209 17621 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
/tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790713339161492-17621-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790713339161492-17621-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790713339161492-17621-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790713339161492-17621-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.53.124:41889
--webserver_interface=127.17.53.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.124.keytab
--principal=kudu/127.17.53.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.53.84:39951
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.53.126:33377,127.17.53.125:43633,127.17.53.124:41889
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790713339161492-17621-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790713339161492-17621-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790713339161492-17621-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 20:28:19.185024 7761 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.53.124" port: 41889 } attempt: 1
I20260929 20:28:19.186314 7764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:28:19.185795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.124:41889 (local address 127.0.0.1:60760)
0929 20:28:19.186001 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0929 20:28:19.186106 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.53.124:41889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260929 20:28:19.187384 7764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:28:19.187038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.124:41889 (local address 127.0.0.1:60770)
0929 20:28:19.187183 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0929 20:28:19.187286 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.53.124:41889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260929 20:28:19.187733 7761 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.17.53.124:41889: Network error: Client connection negotiation failed: client connection to 127.17.53.124:41889: connect: Connection refused (error 111)
I20260929 20:28:19.256021 7761 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.53.124" port: 41889 } attempt: 2
I20260929 20:28:19.257203 7764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:28:19.256747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.124:41889 (local address 127.0.0.1:60772)
0929 20:28:19.256953 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0929 20:28:19.257049 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.53.124:41889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260929 20:28:19.258347 7764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:28:19.258014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.124:41889 (local address 127.0.0.1:60778)
0929 20:28:19.258155 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0929 20:28:19.258224 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.53.124:41889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260929 20:28:19.258621 7761 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.17.53.124:41889: Network error: Client connection negotiation failed: client connection to 127.17.53.124:41889: connect: Connection refused (error 111)
I20260929 20:28:19.276065 7687 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.53.125" port: 43633 } attempt: 5
Sep 29 20:28:19 dist-test-slave-zbnc krb5kdc[7607](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713698, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.53.126@KRBTEST.COM for kudu/127.17.53.125@KRBTEST.COM
I20260929 20:28:19.291349 7689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:28:19.276791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.125:43633 (local address 127.0.0.1:57324)
0929 20:28:19.276975 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0929 20:28:19.276997 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0929 20:28:19.277142 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:28:19.277579 (+ 437us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:28:19.277586 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:28:19.278012 (+ 426us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:28:19.278195 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:28:19.278204 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:28:19.279531 (+ 1327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:28:19.279538 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:28:19.280998 (+ 1460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:28:19.281017 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:28:19.282036 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:28:19.282043 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:28:19.282295 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:28:19.282977 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:28:19.283002 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:28:19.285550 (+ 2548us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:28:19.288875 (+ 3325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:28:19.288881 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:28:19.288886 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:28:19.289268 (+ 382us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:28:19.289653 (+ 385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:28:19.289658 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:28:19.289661 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:28:19.289803 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:28:19.290332 (+ 529us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:28:19.290336 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:28:19.290505 (+ 169us) client_negotiation.cc:769] Sending connection context
0929 20:28:19.290653 (+ 148us) client_negotiation.cc:240] Negotiation successful
0929 20:28:19.290979 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60}
I20260929 20:28:19.291345 7765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:28:19.277039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57324 (local address 127.17.53.125:43633)
0929 20:28:19.277281 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0929 20:28:19.277287 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:28:19.277302 (+ 15us) server_negotiation.cc:407] Connection header received
0929 20:28:19.277353 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:28:19.277356 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:28:19.277428 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:28:19.277537 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:28:19.278357 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:28:19.279189 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:28:19.281377 (+ 2188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:28:19.281863 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:28:19.281961 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:28:19.285753 (+ 3792us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:28:19.285758 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:28:19.285763 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:28:19.285798 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:28:19.288735 (+ 2937us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:28:19.289398 (+ 663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:28:19.289411 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:28:19.289414 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:28:19.289534 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:28:19.289913 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:28:19.289919 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:28:19.289922 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:28:19.290212 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:28:19.290369 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0929 20:28:19.290751 (+ 382us) server_negotiation.cc:299] Negotiation successful
0929 20:28:19.291009 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260929 20:28:19.292546 7687 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.53.124" port: 41889 } has no permanent_uuid. Determining permanent_uuid...
I20260929 20:28:19.293805 7689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:28:19.293406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.124:41889 (local address 127.0.0.1:60786)
0929 20:28:19.293571 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0929 20:28:19.293654 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.53.124:41889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260929 20:28:19.294746 7689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:28:19.294429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.124:41889 (local address 127.0.0.1:60792)
0929 20:28:19.294568 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0929 20:28:19.294641 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.53.124:41889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260929 20:28:19.294977 7687 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.17.53.124:41889: Network error: Client connection negotiation failed: client connection to 127.17.53.124:41889: connect: Connection refused (error 111)
I20260929 20:28:19.346233 7687 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.53.124" port: 41889 } attempt: 1
I20260929 20:28:19.347544 7689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:28:19.346961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.124:41889 (local address 127.0.0.1:60796)
0929 20:28:19.347247 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0929 20:28:19.347369 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.53.124:41889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260929 20:28:19.348644 7689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:28:19.348298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.124:41889 (local address 127.0.0.1:60798)
0929 20:28:19.348448 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0929 20:28:19.348533 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.53.124:41889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260929 20:28:19.348955 7687 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.17.53.124:41889: Network error: Client connection negotiation failed: client connection to 127.17.53.124:41889: connect: Connection refused (error 111)
I20260929 20:28:19.349850 7761 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.53.124" port: 41889 } attempt: 3
I20260929 20:28:19.350919 7764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:28:19.350463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.124:41889 (local address 127.0.0.1:60810)
0929 20:28:19.350653 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0929 20:28:19.350737 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.53.124:41889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260929 20:28:19.351920 7764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:28:19.351517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.124:41889 (local address 127.0.0.1:60826)
0929 20:28:19.351731 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0929 20:28:19.351815 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.53.124:41889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119}
W20260929 20:28:19.352169 7761 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.17.53.124:41889: Network error: Client connection negotiation failed: client connection to 127.17.53.124:41889: connect: Connection refused (error 111)
I20260929 20:28:19.417263 7687 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.53.124" port: 41889 } attempt: 2
I20260929 20:28:19.418537 7689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:28:19.418056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.124:41889 (local address 127.0.0.1:60832)
0929 20:28:19.418276 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0929 20:28:19.418379 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.53.124:41889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260929 20:28:19.419556 7689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:28:19.419222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.124:41889 (local address 127.0.0.1:60848)
0929 20:28:19.419374 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0929 20:28:19.419450 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.53.124:41889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260929 20:28:19.419773 7687 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.17.53.124:41889: Network error: Client connection negotiation failed: client connection to 127.17.53.124:41889: connect: Connection refused (error 111)
W20260929 20:28:19.495281 7768 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
I20260929 20:28:19.495424 7761 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.53.124" port: 41889 } attempt: 4
W20260929 20:28:19.495687 7768 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:28:19.495833 7768 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:28:19.496006 7768 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260929 20:28:19.496598 7764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:28:19.496135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.124:41889 (local address 127.0.0.1:60858)
0929 20:28:19.496370 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0929 20:28:19.496461 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.53.124:41889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260929 20:28:19.497622 7764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:28:19.497351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.124:41889 (local address 127.0.0.1:60874)
0929 20:28:19.497462 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0929 20:28:19.497530 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.53.124:41889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260929 20:28:19.497876 7761 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.17.53.124:41889: Network error: Client connection negotiation failed: client connection to 127.17.53.124:41889: connect: Connection refused (error 111)
W20260929 20:28:19.503925 7768 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:28:19.504009 7768 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:28:19.504151 7768 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:28:19.504232 7768 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:28:19.504336 7768 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:28:19.514550 7768 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.53.84:39951
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790713339161492-17621-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790713339161492-17621-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.17.53.126:33377,127.17.53.125:43633,127.17.53.124:41889
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.124.keytab
--principal=kudu/127.17.53.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.53.124:41889
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790713339161492-17621-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.17.53.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790713339161492-17621-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:28:19.515935 7768 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:28:19.517261 7768 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 20:28:19.525964 7687 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.53.124" port: 41889 } attempt: 3
I20260929 20:28:19.527148 7689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:28:19.526609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.124:41889 (local address 127.0.0.1:60884)
0929 20:28:19.526849 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0929 20:28:19.526984 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.53.124:41889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260929 20:28:19.528690 7689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:28:19.528296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.124:41889 (local address 127.0.0.1:60888)
0929 20:28:19.528503 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0929 20:28:19.528578 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.53.124:41889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
W20260929 20:28:19.528903 7687 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.17.53.124:41889: Network error: Client connection negotiation failed: client connection to 127.17.53.124:41889: connect: Connection refused (error 111)
W20260929 20:28:19.531004 7776 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:28:19.531039 7774 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:28:19.531199 7773 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:28:19.532732 7768 server_base.cc:1034] running on GCE node
I20260929 20:28:19.533682 7768 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:28:19.535321 7768 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:28:19.536587 7768 hybrid_clock.cc:648] HybridClock initialized: now 1790713699536509 us; error 50 us; skew 500 ppm
I20260929 20:28:19.536966 7768 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:28:19 dist-test-slave-zbnc krb5kdc[7607](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713699, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.53.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:28:19.540344 7768 init.cc:377] Logged in from keytab as kudu/127.17.53.124@KRBTEST.COM (short username kudu)
I20260929 20:28:19.541935 7768 webserver.cc:460] Webserver started at http://127.17.53.124:36315/ using document root <none> and password file <none>
I20260929 20:28:19.542678 7768 fs_manager.cc:359] Metadata directory not provided
I20260929 20:28:19.542827 7768 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:28:19.543200 7768 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:28:19.546299 7768 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790713339161492-17621-0/minicluster-data/master-2/data/instance:
uuid: "27e2b4963a95424090854f955bf1e1d2"
format_stamp: "Formatted at 2026-09-29 20:28:19 on dist-test-slave-zbnc"
server_key: "c89b9aab83c341696fb2b2e744040f0f"
server_key_iv: "892f723eca55221f9e967775b324d4be"
server_key_version: "encryptionkey@0"
I20260929 20:28:19.547501 7768 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790713339161492-17621-0/minicluster-data/master-2/wal/instance:
uuid: "27e2b4963a95424090854f955bf1e1d2"
format_stamp: "Formatted at 2026-09-29 20:28:19 on dist-test-slave-zbnc"
server_key: "c89b9aab83c341696fb2b2e744040f0f"
server_key_iv: "892f723eca55221f9e967775b324d4be"
server_key_version: "encryptionkey@0"
I20260929 20:28:19.553853 7768 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.002s
I20260929 20:28:19.557713 7783 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:28:19.559139 7768 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 20:28:19.559329 7768 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790713339161492-17621-0/minicluster-data/master-2/data,/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790713339161492-17621-0/minicluster-data/master-2/wal
uuid: "27e2b4963a95424090854f955bf1e1d2"
format_stamp: "Formatted at 2026-09-29 20:28:19 on dist-test-slave-zbnc"
server_key: "c89b9aab83c341696fb2b2e744040f0f"
server_key_iv: "892f723eca55221f9e967775b324d4be"
server_key_version: "encryptionkey@0"
I20260929 20:28:19.559563 7768 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790713339161492-17621-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790713339161492-17621-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790713339161492-17621-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:28:19.591034 7768 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:28:19.595587 7768 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:28:19.595883 7768 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:28:19.614455 7768 rpc_server.cc:307] RPC server started. Bound to: 127.17.53.124:41889
I20260929 20:28:19.614477 7834 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.53.124:41889 every 8 connection(s)
I20260929 20:28:19.615936 7768 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790713339161492-17621-0/minicluster-data/master-2/data/info.pb
I20260929 20:28:19.619619 7835 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:28:19.624666 17621 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu as pid 7768
I20260929 20:28:19.624869 17621 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790713339161492-17621-0/minicluster-data/master-2/wal/instance
I20260929 20:28:19.625337 17621 external_mini_cluster.cc:1428] Setting key e2b1b081a9e96b43459898cd6e2e2525
I20260929 20:28:19.626222 7835 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.53.126" port: 33377 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 20:28:19 dist-test-slave-zbnc krb5kdc[7607](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713699, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.53.124@KRBTEST.COM for kudu/127.17.53.126@KRBTEST.COM
Sep 29 20:28:19 dist-test-slave-zbnc krb5kdc[7607](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713698, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.53.124@KRBTEST.COM
I20260929 20:28:19.647444 7839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:28:19.628974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45116 (local address 127.17.53.126:33377)
0929 20:28:19.629627 (+ 653us) server_negotiation.cc:206] Beginning negotiation
0929 20:28:19.629637 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 20:28:19.630188 (+ 551us) server_negotiation.cc:407] Connection header received
0929 20:28:19.630915 (+ 727us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:28:19.630922 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:28:19.631023 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:28:19.631168 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:28:19.632519 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:28:19.633474 (+ 955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:28:19.635298 (+ 1824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:28:19.636028 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:28:19.636141 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:28:19.640429 (+ 4288us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:28:19.640439 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:28:19.640445 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:28:19.640500 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:28:19.643773 (+ 3273us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:28:19.644707 (+ 934us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:28:19.644711 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:28:19.644713 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:28:19.644849 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:28:19.645330 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:28:19.645335 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:28:19.645339 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:28:19.645725 (+ 386us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:28:19.645924 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0929 20:28:19.646807 (+ 883us) server_negotiation.cc:299] Negotiation successful
0929 20:28:19.647028 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":491,"thread_start_us":256,"threads_started":1}
I20260929 20:28:19.648747 7840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:28:19.628977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:33377 (local address 127.0.0.1:45116)
0929 20:28:19.629962 (+ 985us) negotiation.cc:107] Waiting for socket to connect
0929 20:28:19.630008 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0929 20:28:19.630619 (+ 611us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:28:19.631238 (+ 619us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:28:19.631247 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:28:19.631977 (+ 730us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:28:19.632331 (+ 354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:28:19.632362 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:28:19.633637 (+ 1275us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:28:19.633643 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:28:19.635154 (+ 1511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:28:19.635166 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:28:19.636279 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:28:19.636286 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:28:19.636890 (+ 604us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:28:19.637733 (+ 843us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:28:19.637753 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:28:19.640197 (+ 2444us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:28:19.643900 (+ 3703us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:28:19.643911 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:28:19.643925 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:28:19.644518 (+ 593us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:28:19.644970 (+ 452us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:28:19.644975 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:28:19.644979 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:28:19.645206 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:28:19.645909 (+ 703us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:28:19.645937 (+ 28us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:28:19.646383 (+ 446us) client_negotiation.cc:769] Sending connection context
0929 20:28:19.646719 (+ 336us) client_negotiation.cc:240] Negotiation successful
0929 20:28:19.647238 (+ 519us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":727,"mutex_wait_us":73,"thread_start_us":541,"threads_started":1}
I20260929 20:28:19.648768 7838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:28:19.627338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60890 (local address 127.17.53.124:41889)
0929 20:28:19.628099 (+ 761us) server_negotiation.cc:206] Beginning negotiation
0929 20:28:19.628110 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0929 20:28:19.628149 (+ 39us) server_negotiation.cc:407] Connection header received
0929 20:28:19.630594 (+ 2445us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:28:19.630615 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:28:19.630912 (+ 297us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:28:19.631134 (+ 222us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:28:19.632872 (+ 1738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:28:19.634336 (+ 1464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:28:19.636041 (+ 1705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:28:19.637134 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:28:19.637298 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:28:19.641969 (+ 4671us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:28:19.641977 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:28:19.641989 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:28:19.642017 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:28:19.645143 (+ 3126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:28:19.645990 (+ 847us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:28:19.645997 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:28:19.646000 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:28:19.646141 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:28:19.646535 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:28:19.646541 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:28:19.646543 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:28:19.647001 (+ 458us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:28:19.647294 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0929 20:28:19.647644 (+ 350us) server_negotiation.cc:299] Negotiation successful
0929 20:28:19.647886 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":110,"server-negotiator.queue_time_us":551,"thread_start_us":350,"threads_started":1}
I20260929 20:28:19.651273 7835 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.53.125" port: 43633 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 20:28:19 dist-test-slave-zbnc krb5kdc[7607](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713699, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.53.124@KRBTEST.COM for kudu/127.17.53.125@KRBTEST.COM
Sep 29 20:28:19 dist-test-slave-zbnc krb5kdc[7607](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713698, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.53.126@KRBTEST.COM
I20260929 20:28:19.666654 7840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:28:19.652153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.125:43633 (local address 127.0.0.1:57328)
0929 20:28:19.652313 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0929 20:28:19.652330 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0929 20:28:19.652437 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:28:19.653055 (+ 618us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:28:19.653060 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:28:19.653433 (+ 373us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:28:19.653596 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:28:19.653605 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:28:19.654976 (+ 1371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:28:19.654980 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:28:19.656314 (+ 1334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:28:19.656330 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:28:19.657193 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:28:19.657210 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:28:19.657391 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:28:19.658057 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:28:19.658088 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:28:19.660657 (+ 2569us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:28:19.664125 (+ 3468us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:28:19.664132 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:28:19.664136 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:28:19.664622 (+ 486us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:28:19.665032 (+ 410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:28:19.665037 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:28:19.665041 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:28:19.665191 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:28:19.665719 (+ 528us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:28:19.665724 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:28:19.665871 (+ 147us) client_negotiation.cc:769] Sending connection context
0929 20:28:19.666013 (+ 142us) client_negotiation.cc:240] Negotiation successful
0929 20:28:19.666286 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20260929 20:28:19.666678 7765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:28:19.652495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57328 (local address 127.17.53.125:43633)
0929 20:28:19.652726 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0929 20:28:19.652733 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:28:19.652754 (+ 21us) server_negotiation.cc:407] Connection header received
0929 20:28:19.652816 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:28:19.652822 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:28:19.652898 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:28:19.653035 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:28:19.653751 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:28:19.654825 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:28:19.656465 (+ 1640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:28:19.657018 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:28:19.657150 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:28:19.660862 (+ 3712us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:28:19.660867 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:28:19.660870 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:28:19.660907 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:28:19.663994 (+ 3087us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:28:19.664758 (+ 764us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:28:19.664764 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:28:19.664767 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:28:19.664912 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:28:19.665302 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:28:19.665308 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:28:19.665311 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:28:19.665566 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:28:19.665732 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0929 20:28:19.666101 (+ 369us) server_negotiation.cc:299] Negotiation successful
0929 20:28:19.666316 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260929 20:28:19.667852 7835 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.53.124" port: 41889 } has no permanent_uuid. Determining permanent_uuid...
I20260929 20:28:19.668481 7839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:28:19.651983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45120 (local address 127.17.53.126:33377)
0929 20:28:19.652151 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0929 20:28:19.652157 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:28:19.652291 (+ 134us) server_negotiation.cc:407] Connection header received
0929 20:28:19.652501 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:28:19.652508 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:28:19.652630 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:28:19.652735 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:28:19.653986 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:28:19.655099 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:28:19.656736 (+ 1637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:28:19.657375 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:28:19.657464 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:28:19.662152 (+ 4688us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:28:19.662157 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:28:19.662160 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:28:19.662189 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:28:19.665389 (+ 3200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:28:19.666093 (+ 704us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:28:19.666105 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:28:19.666108 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:28:19.666229 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:28:19.666633 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:28:19.666639 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:28:19.666643 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:28:19.667097 (+ 454us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:28:19.667351 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0929 20:28:19.667943 (+ 592us) server_negotiation.cc:299] Negotiation successful
0929 20:28:19.668170 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
Sep 29 20:28:19 dist-test-slave-zbnc krb5kdc[7607](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713699, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.53.124@KRBTEST.COM for kudu/127.17.53.124@KRBTEST.COM
Sep 29 20:28:19 dist-test-slave-zbnc krb5kdc[7607](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713698, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.53.125@KRBTEST.COM
I20260929 20:28:19.683320 7840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:28:19.668752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.124:41889 (local address 127.0.0.1:60898)
0929 20:28:19.669486 (+ 734us) negotiation.cc:107] Waiting for socket to connect
0929 20:28:19.669507 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 20:28:19.669634 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:28:19.670018 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:28:19.670026 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:28:19.670507 (+ 481us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:28:19.670690 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:28:19.670704 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:28:19.672097 (+ 1393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:28:19.672101 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:28:19.673416 (+ 1315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:28:19.673429 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:28:19.674359 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:28:19.674363 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:28:19.674532 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:28:19.675194 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:28:19.675211 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:28:19.677287 (+ 2076us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:28:19.680720 (+ 3433us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:28:19.680724 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:28:19.680727 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:28:19.681096 (+ 369us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:28:19.681523 (+ 427us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:28:19.681527 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:28:19.681530 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:28:19.681735 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:28:19.682404 (+ 669us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:28:19.682410 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:28:19.682588 (+ 178us) client_negotiation.cc:769] Sending connection context
0929 20:28:19.682719 (+ 131us) client_negotiation.cc:240] Negotiation successful
0929 20:28:19.683040 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149}
I20260929 20:28:19.683665 7838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:28:19.668701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60898 (local address 127.17.53.124:41889)
0929 20:28:19.668904 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0929 20:28:19.668910 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:28:19.669699 (+ 789us) server_negotiation.cc:407] Connection header received
0929 20:28:19.669765 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:28:19.669769 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:28:19.669839 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:28:19.669947 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:28:19.670836 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:28:19.671972 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:28:19.673584 (+ 1612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:28:19.674206 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:28:19.674340 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:28:19.677495 (+ 3155us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:28:19.677504 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:28:19.677510 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:28:19.677550 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:28:19.680585 (+ 3035us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:28:19.681240 (+ 655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:28:19.681246 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:28:19.681249 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:28:19.681391 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:28:19.681859 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:28:19.681865 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:28:19.681868 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:28:19.682253 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:28:19.682393 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0929 20:28:19.682901 (+ 508us) server_negotiation.cc:299] Negotiation successful
0929 20:28:19.683272 (+ 371us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":65,"server-negotiator.queue_time_us":74}
I20260929 20:28:19.687047 7765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:28:19.670327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57340 (local address 127.17.53.125:43633)
0929 20:28:19.670518 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0929 20:28:19.670523 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:28:19.670540 (+ 17us) server_negotiation.cc:407] Connection header received
0929 20:28:19.670587 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:28:19.670591 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:28:19.670660 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:28:19.670800 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:28:19.672355 (+ 1555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:28:19.673406 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:28:19.675186 (+ 1780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:28:19.675607 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:28:19.675745 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:28:19.680656 (+ 4911us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:28:19.680665 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:28:19.680673 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:28:19.680763 (+ 90us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:28:19.684086 (+ 3323us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:28:19.685022 (+ 936us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:28:19.685026 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:28:19.685028 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:28:19.685162 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:28:19.685586 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:28:19.685591 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:28:19.685599 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:28:19.685865 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:28:19.686147 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0929 20:28:19.686442 (+ 295us) server_negotiation.cc:299] Negotiation successful
0929 20:28:19.686680 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260929 20:28:19.689384 7835 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 27e2b4963a95424090854f955bf1e1d2: Bootstrap starting.
I20260929 20:28:19.692356 17621 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 20:28:19.693391 7835 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 27e2b4963a95424090854f955bf1e1d2: Neither blocks nor log segments found. Creating new log.
I20260929 20:28:19.694530 7835 log.cc:824] T 00000000000000000000000000000000 P 27e2b4963a95424090854f955bf1e1d2: Log is configured to *not* fsync() on all Append() calls
I20260929 20:28:19.697484 7835 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 27e2b4963a95424090854f955bf1e1d2: No bootstrap required, opened a new log
I20260929 20:28:19.695276 7662 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "3f9b2f0ee39b4efa93370234003a35d0"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45120
I20260929 20:28:19.704519 7835 raft_consensus.cc:348] T 00000000000000000000000000000000 P 27e2b4963a95424090854f955bf1e1d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f9b2f0ee39b4efa93370234003a35d0" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 33377 } } peers { permanent_uuid: "200282175f6d4993a9937351cf946aeb" member_type: VOTER last_known_addr { host: "127.17.53.125" port: 43633 } } peers { permanent_uuid: "27e2b4963a95424090854f955bf1e1d2" member_type: VOTER last_known_addr { host: "127.17.53.124" port: 41889 } }
I20260929 20:28:19.704846 7835 raft_consensus.cc:729] T 00000000000000000000000000000000 P 27e2b4963a95424090854f955bf1e1d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 27e2b4963a95424090854f955bf1e1d2, State: Initialized, Role: FOLLOWER
I20260929 20:28:19.705300 7835 consensus_queue.cc:260] T 00000000000000000000000000000000 P 27e2b4963a95424090854f955bf1e1d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f9b2f0ee39b4efa93370234003a35d0" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 33377 } } peers { permanent_uuid: "200282175f6d4993a9937351cf946aeb" member_type: VOTER last_known_addr { host: "127.17.53.125" port: 43633 } } peers { permanent_uuid: "27e2b4963a95424090854f955bf1e1d2" member_type: VOTER last_known_addr { host: "127.17.53.124" port: 41889 } }
I20260929 20:28:19.706177 7687 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.53.124" port: 41889 } attempt: 4
I20260929 20:28:19.706308 7842 sys_catalog.cc:455] T 00000000000000000000000000000000 P 27e2b4963a95424090854f955bf1e1d2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f9b2f0ee39b4efa93370234003a35d0" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 33377 } } peers { permanent_uuid: "200282175f6d4993a9937351cf946aeb" member_type: VOTER last_known_addr { host: "127.17.53.125" port: 43633 } } peers { permanent_uuid: "27e2b4963a95424090854f955bf1e1d2" member_type: VOTER last_known_addr { host: "127.17.53.124" port: 41889 } } }
I20260929 20:28:19.706879 7842 sys_catalog.cc:458] T 00000000000000000000000000000000 P 27e2b4963a95424090854f955bf1e1d2 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 20:28:19.707659 7835 sys_catalog.cc:564] T 00000000000000000000000000000000 P 27e2b4963a95424090854f955bf1e1d2 [sys.catalog]: configured and running, proceeding with master startup.
Sep 29 20:28:19 dist-test-slave-zbnc krb5kdc[7607](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713698, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.53.126@KRBTEST.COM for kudu/127.17.53.124@KRBTEST.COM
W20260929 20:28:19.717898 7853 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 27e2b4963a95424090854f955bf1e1d2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 20:28:19.718097 7853 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 20:28:19.723122 7689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:28:19.706874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.124:41889 (local address 127.0.0.1:60912)
0929 20:28:19.707153 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0929 20:28:19.707169 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 20:28:19.707276 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:28:19.707625 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:28:19.707629 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:28:19.708123 (+ 494us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:28:19.708330 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:28:19.708341 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:28:19.710053 (+ 1712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:28:19.710059 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:28:19.711429 (+ 1370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:28:19.711449 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:28:19.712729 (+ 1280us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:28:19.712736 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:28:19.712942 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:28:19.713578 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:28:19.713606 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:28:19.716591 (+ 2985us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:28:19.720346 (+ 3755us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:28:19.720354 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:28:19.720361 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:28:19.720747 (+ 386us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:28:19.721177 (+ 430us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:28:19.721183 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:28:19.721188 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:28:19.721375 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:28:19.722101 (+ 726us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:28:19.722106 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:28:19.722270 (+ 164us) client_negotiation.cc:769] Sending connection context
0929 20:28:19.722414 (+ 144us) client_negotiation.cc:240] Negotiation successful
0929 20:28:19.722705 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159}
I20260929 20:28:19.724025 7838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:28:19.706964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60912 (local address 127.17.53.124:41889)
0929 20:28:19.707156 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0929 20:28:19.707163 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:28:19.707253 (+ 90us) server_negotiation.cc:407] Connection header received
0929 20:28:19.707393 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:28:19.707398 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:28:19.707472 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:28:19.707583 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:28:19.708492 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:28:19.709881 (+ 1389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:28:19.711604 (+ 1723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:28:19.712554 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:28:19.712707 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:28:19.716829 (+ 4122us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:28:19.716835 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:28:19.716840 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:28:19.716875 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:28:19.720169 (+ 3294us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:28:19.720890 (+ 721us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:28:19.720896 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:28:19.720899 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:28:19.721047 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:28:19.721502 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:28:19.721506 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:28:19.721515 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:28:19.721950 (+ 435us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:28:19.722136 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0929 20:28:19.722525 (+ 389us) server_negotiation.cc:299] Negotiation successful
0929 20:28:19.722753 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 20:28:19.715823 7768 master_runner.cc:186] Error getting master registration for 127.17.53.126:33377: OK, instance_id { permanent_uuid: "3f9b2f0ee39b4efa93370234003a35d0" instance_seqno: 1790713698565425 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 20:28:19.728296 7768 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 20:28:19.729702 7687 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f9b2f0ee39b4efa93370234003a35d0: Bootstrap starting.
I20260929 20:28:19.733613 7687 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3f9b2f0ee39b4efa93370234003a35d0: Neither blocks nor log segments found. Creating new log.
I20260929 20:28:19.734745 7687 log.cc:824] T 00000000000000000000000000000000 P 3f9b2f0ee39b4efa93370234003a35d0: Log is configured to *not* fsync() on all Append() calls
I20260929 20:28:19.737491 7687 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3f9b2f0ee39b4efa93370234003a35d0: No bootstrap required, opened a new log
I20260929 20:28:19.745205 7687 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3f9b2f0ee39b4efa93370234003a35d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f9b2f0ee39b4efa93370234003a35d0" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 33377 } } peers { permanent_uuid: "200282175f6d4993a9937351cf946aeb" member_type: VOTER last_known_addr { host: "127.17.53.125" port: 43633 } } peers { permanent_uuid: "27e2b4963a95424090854f955bf1e1d2" member_type: VOTER last_known_addr { host: "127.17.53.124" port: 41889 } }
I20260929 20:28:19.745668 7687 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3f9b2f0ee39b4efa93370234003a35d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f9b2f0ee39b4efa93370234003a35d0, State: Initialized, Role: FOLLOWER
I20260929 20:28:19.746150 7687 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3f9b2f0ee39b4efa93370234003a35d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f9b2f0ee39b4efa93370234003a35d0" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 33377 } } peers { permanent_uuid: "200282175f6d4993a9937351cf946aeb" member_type: VOTER last_known_addr { host: "127.17.53.125" port: 43633 } } peers { permanent_uuid: "27e2b4963a95424090854f955bf1e1d2" member_type: VOTER last_known_addr { host: "127.17.53.124" port: 41889 } }
I20260929 20:28:19.747184 7856 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f9b2f0ee39b4efa93370234003a35d0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f9b2f0ee39b4efa93370234003a35d0" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 33377 } } peers { permanent_uuid: "200282175f6d4993a9937351cf946aeb" member_type: VOTER last_known_addr { host: "127.17.53.125" port: 43633 } } peers { permanent_uuid: "27e2b4963a95424090854f955bf1e1d2" member_type: VOTER last_known_addr { host: "127.17.53.124" port: 41889 } } }
I20260929 20:28:19.747645 7856 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f9b2f0ee39b4efa93370234003a35d0 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 20:28:19.748519 7687 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3f9b2f0ee39b4efa93370234003a35d0 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:28:19.748725 7662 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3f9b2f0ee39b4efa93370234003a35d0 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260929 20:28:19.749253 7662 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3f9b2f0ee39b4efa93370234003a35d0 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:28:19.753391 7662 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3f9b2f0ee39b4efa93370234003a35d0 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f9b2f0ee39b4efa93370234003a35d0" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 33377 } } peers { permanent_uuid: "200282175f6d4993a9937351cf946aeb" member_type: VOTER last_known_addr { host: "127.17.53.125" port: 43633 } } peers { permanent_uuid: "27e2b4963a95424090854f955bf1e1d2" member_type: VOTER last_known_addr { host: "127.17.53.124" port: 41889 } }
I20260929 20:28:19.755934 7810 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3f9b2f0ee39b4efa93370234003a35d0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "27e2b4963a95424090854f955bf1e1d2"
I20260929 20:28:19.756461 7810 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 27e2b4963a95424090854f955bf1e1d2 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:28:19.760079 7810 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 27e2b4963a95424090854f955bf1e1d2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f9b2f0ee39b4efa93370234003a35d0 in term 1.
I20260929 20:28:19.760676 7662 leader_election.cc:288] T 00000000000000000000000000000000 P 3f9b2f0ee39b4efa93370234003a35d0 [CANDIDATE]: Term 1 election: Requested vote from peers 200282175f6d4993a9937351cf946aeb (127.17.53.125:43633), 27e2b4963a95424090854f955bf1e1d2 (127.17.53.124:41889)
I20260929 20:28:19.760921 7639 leader_election.cc:302] T 00000000000000000000000000000000 P 3f9b2f0ee39b4efa93370234003a35d0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 27e2b4963a95424090854f955bf1e1d2, 3f9b2f0ee39b4efa93370234003a35d0; no voters:
I20260929 20:28:19.761756 7856 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3f9b2f0ee39b4efa93370234003a35d0 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:28:19.770884 7856 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3f9b2f0ee39b4efa93370234003a35d0 [term 1 LEADER]: Becoming Leader. State: Replica: 3f9b2f0ee39b4efa93370234003a35d0, State: Running, Role: LEADER
I20260929 20:28:19.772615 7856 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3f9b2f0ee39b4efa93370234003a35d0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f9b2f0ee39b4efa93370234003a35d0" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 33377 } } peers { permanent_uuid: "200282175f6d4993a9937351cf946aeb" member_type: VOTER last_known_addr { host: "127.17.53.125" port: 43633 } } peers { permanent_uuid: "27e2b4963a95424090854f955bf1e1d2" member_type: VOTER last_known_addr { host: "127.17.53.124" port: 41889 } }
I20260929 20:28:19.787750 7859 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f9b2f0ee39b4efa93370234003a35d0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3f9b2f0ee39b4efa93370234003a35d0. Latest consensus state: current_term: 1 leader_uuid: "3f9b2f0ee39b4efa93370234003a35d0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f9b2f0ee39b4efa93370234003a35d0" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 33377 } } peers { permanent_uuid: "200282175f6d4993a9937351cf946aeb" member_type: VOTER last_known_addr { host: "127.17.53.125" port: 43633 } } peers { permanent_uuid: "27e2b4963a95424090854f955bf1e1d2" member_type: VOTER last_known_addr { host: "127.17.53.124" port: 41889 } } }
I20260929 20:28:19.788371 7859 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f9b2f0ee39b4efa93370234003a35d0 [sys.catalog]: This master's current role is: LEADER
I20260929 20:28:19.797083 7761 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.53.124" port: 41889 } attempt: 5
I20260929 20:28:19.800835 7868 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
W20260929 20:28:19.804162 7641 tablet.cc:2369] T 00000000000000000000000000000000 P 3f9b2f0ee39b4efa93370234003a35d0: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 29 20:28:19 dist-test-slave-zbnc krb5kdc[7607](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713699, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.53.125@KRBTEST.COM for kudu/127.17.53.124@KRBTEST.COM
I20260929 20:28:19.818738 7764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:28:19.797790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.124:41889 (local address 127.0.0.1:60914)
0929 20:28:19.798231 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0929 20:28:19.798255 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0929 20:28:19.798394 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:28:19.799533 (+ 1139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:28:19.799541 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:28:19.800199 (+ 658us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:28:19.800473 (+ 274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:28:19.800489 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:28:19.802285 (+ 1796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:28:19.802293 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:28:19.804088 (+ 1795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:28:19.804112 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:28:19.805665 (+ 1553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:28:19.805674 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:28:19.806004 (+ 330us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:28:19.806897 (+ 893us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:28:19.806932 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:28:19.810516 (+ 3584us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:28:19.815483 (+ 4967us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:28:19.815491 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:28:19.815498 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:28:19.816149 (+ 651us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:28:19.816653 (+ 504us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:28:19.816660 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:28:19.816666 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:28:19.816862 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:28:19.817647 (+ 785us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:28:19.817655 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:28:19.817859 (+ 204us) client_negotiation.cc:769] Sending connection context
0929 20:28:19.818020 (+ 161us) client_negotiation.cc:240] Negotiation successful
0929 20:28:19.818314 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281}
I20260929 20:28:19.819772 7838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:28:19.798632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60914 (local address 127.17.53.124:41889)
0929 20:28:19.798899 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0929 20:28:19.798908 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 20:28:19.798935 (+ 27us) server_negotiation.cc:407] Connection header received
0929 20:28:19.799019 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:28:19.799027 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:28:19.799168 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:28:19.799297 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:28:19.800713 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:28:19.802087 (+ 1374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:28:19.804316 (+ 2229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:28:19.805264 (+ 948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:28:19.805414 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:28:19.810757 (+ 5343us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:28:19.810767 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:28:19.810775 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:28:19.810822 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:28:19.815267 (+ 4445us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:28:19.816310 (+ 1043us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:28:19.816319 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:28:19.816325 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:28:19.816506 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:28:19.817004 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:28:19.817011 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:28:19.817017 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:28:19.817417 (+ 400us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:28:19.817578 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0929 20:28:19.819126 (+ 1548us) server_negotiation.cc:299] Negotiation successful
0929 20:28:19.819366 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260929 20:28:19.827711 7868 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:28:19.833963 7761 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 200282175f6d4993a9937351cf946aeb: Bootstrap starting.
I20260929 20:28:19.837229 7879 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260929 20:28:19.838120 7761 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 200282175f6d4993a9937351cf946aeb: Neither blocks nor log segments found. Creating new log.
I20260929 20:28:19.842307 7761 log.cc:824] T 00000000000000000000000000000000 P 200282175f6d4993a9937351cf946aeb: Log is configured to *not* fsync() on all Append() calls
W20260929 20:28:19.855137 7789 tablet.cc:2369] T 00000000000000000000000000000000 P 27e2b4963a95424090854f955bf1e1d2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 20:28:19.841763 7620 master_runner.cc:186] Error getting master registration for 127.17.53.125:43633: OK, instance_id { permanent_uuid: "200282175f6d4993a9937351cf946aeb" instance_seqno: 1790713699069843 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 20:28:19.859249 7620 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 20:28:19.860250 7839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:28:19.786937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45130 (local address 127.17.53.126:33377)
0929 20:28:19.787163 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0929 20:28:19.787169 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:28:19.818291 (+ 31122us) server_negotiation.cc:407] Connection header received
0929 20:28:19.818585 (+ 294us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:28:19.818591 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:28:19.818671 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:28:19.818749 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:28:19.822244 (+ 3495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:28:19.823478 (+ 1234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:28:19.843540 (+ 20062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:28:19.844302 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:28:19.844416 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:28:19.850049 (+ 5633us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:28:19.850057 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:28:19.850067 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:28:19.850115 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:28:19.854041 (+ 3926us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:28:19.855583 (+ 1542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:28:19.855591 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:28:19.855594 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:28:19.855862 (+ 268us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:28:19.856611 (+ 749us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:28:19.856616 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:28:19.856620 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:28:19.857260 (+ 640us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:28:19.857715 (+ 455us) server_negotiation.cc:1035] Waiting for connection context
0929 20:28:19.858772 (+ 1057us) server_negotiation.cc:299] Negotiation successful
0929 20:28:19.859221 (+ 449us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260929 20:28:19.864023 7761 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 200282175f6d4993a9937351cf946aeb: No bootstrap required, opened a new log
I20260929 20:28:19.867883 7765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:28:19.791325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57348 (local address 127.17.53.125:43633)
0929 20:28:19.791505 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0929 20:28:19.791511 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:28:19.826313 (+ 34802us) server_negotiation.cc:407] Connection header received
0929 20:28:19.826555 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:28:19.826563 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:28:19.826643 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:28:19.826708 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:28:19.845043 (+ 18335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:28:19.847468 (+ 2425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:28:19.849630 (+ 2162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:28:19.850338 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:28:19.850434 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:28:19.859695 (+ 9261us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:28:19.859715 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:28:19.859720 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:28:19.859758 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:28:19.863364 (+ 3606us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:28:19.864885 (+ 1521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:28:19.864892 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:28:19.864896 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:28:19.865075 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:28:19.865727 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:28:19.865735 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:28:19.865738 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:28:19.866111 (+ 373us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:28:19.866351 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0929 20:28:19.867272 (+ 921us) server_negotiation.cc:299] Negotiation successful
0929 20:28:19.867512 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260929 20:28:19.881114 7810 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 27e2b4963a95424090854f955bf1e1d2 [term 1 FOLLOWER]: Refusing update from remote peer 3f9b2f0ee39b4efa93370234003a35d0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:28:19.882258 7859 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 3f9b2f0ee39b4efa93370234003a35d0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "27e2b4963a95424090854f955bf1e1d2" member_type: VOTER last_known_addr { host: "127.17.53.124" port: 41889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:28:19.883147 7838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:28:19.802827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60924 (local address 127.17.53.124:41889)
0929 20:28:19.820018 (+ 17191us) server_negotiation.cc:206] Beginning negotiation
0929 20:28:19.820024 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:28:19.838940 (+ 18916us) server_negotiation.cc:407] Connection header received
0929 20:28:19.841000 (+ 2060us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:28:19.841007 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:28:19.841137 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:28:19.841230 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:28:19.852509 (+ 11279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:28:19.853585 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:28:19.855831 (+ 2246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:28:19.856508 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:28:19.856604 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:28:19.868571 (+ 11967us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:28:19.868578 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:28:19.868584 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:28:19.868626 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:28:19.873358 (+ 4732us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:28:19.874193 (+ 835us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:28:19.874240 (+ 47us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:28:19.874245 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:28:19.874405 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:28:19.874922 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:28:19.874931 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:28:19.874935 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:28:19.875291 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:28:19.875549 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0929 20:28:19.882365 (+ 6816us) server_negotiation.cc:299] Negotiation successful
0929 20:28:19.882685 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17070,"thread_start_us":25192,"threads_started":1}
I20260929 20:28:19.902387 7842 sys_catalog.cc:455] T 00000000000000000000000000000000 P 27e2b4963a95424090854f955bf1e1d2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3f9b2f0ee39b4efa93370234003a35d0. Latest consensus state: current_term: 1 leader_uuid: "3f9b2f0ee39b4efa93370234003a35d0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f9b2f0ee39b4efa93370234003a35d0" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 33377 } } peers { permanent_uuid: "200282175f6d4993a9937351cf946aeb" member_type: VOTER last_known_addr { host: "127.17.53.125" port: 43633 } } peers { permanent_uuid: "27e2b4963a95424090854f955bf1e1d2" member_type: VOTER last_known_addr { host: "127.17.53.124" port: 41889 } } }
I20260929 20:28:19.902707 7842 sys_catalog.cc:458] T 00000000000000000000000000000000 P 27e2b4963a95424090854f955bf1e1d2 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 20:28:19.903774 7761 raft_consensus.cc:348] T 00000000000000000000000000000000 P 200282175f6d4993a9937351cf946aeb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f9b2f0ee39b4efa93370234003a35d0" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 33377 } } peers { permanent_uuid: "200282175f6d4993a9937351cf946aeb" member_type: VOTER last_known_addr { host: "127.17.53.125" port: 43633 } } peers { permanent_uuid: "27e2b4963a95424090854f955bf1e1d2" member_type: VOTER last_known_addr { host: "127.17.53.124" port: 41889 } }
I20260929 20:28:19.904549 7761 raft_consensus.cc:729] T 00000000000000000000000000000000 P 200282175f6d4993a9937351cf946aeb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 200282175f6d4993a9937351cf946aeb, State: Initialized, Role: FOLLOWER
I20260929 20:28:19.905493 7761 consensus_queue.cc:260] T 00000000000000000000000000000000 P 200282175f6d4993a9937351cf946aeb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f9b2f0ee39b4efa93370234003a35d0" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 33377 } } peers { permanent_uuid: "200282175f6d4993a9937351cf946aeb" member_type: VOTER last_known_addr { host: "127.17.53.125" port: 43633 } } peers { permanent_uuid: "27e2b4963a95424090854f955bf1e1d2" member_type: VOTER last_known_addr { host: "127.17.53.124" port: 41889 } }
I20260929 20:28:19.907291 7882 mvcc.cc:205] Tried to move back new op lower bound from 7334763314655326208 to 7334763314287558656. Current Snapshot: MvccSnapshot[applied={T|T < 7334763314655326208}]
I20260929 20:28:19.910527 7859 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f9b2f0ee39b4efa93370234003a35d0 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3f9b2f0ee39b4efa93370234003a35d0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f9b2f0ee39b4efa93370234003a35d0" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 33377 } } peers { permanent_uuid: "200282175f6d4993a9937351cf946aeb" member_type: VOTER last_known_addr { host: "127.17.53.125" port: 43633 } } peers { permanent_uuid: "27e2b4963a95424090854f955bf1e1d2" member_type: VOTER last_known_addr { host: "127.17.53.124" port: 41889 } } }
I20260929 20:28:19.910818 7859 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f9b2f0ee39b4efa93370234003a35d0 [sys.catalog]: This master's current role is: LEADER
I20260929 20:28:19.919471 7885 sys_catalog.cc:455] T 00000000000000000000000000000000 P 200282175f6d4993a9937351cf946aeb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f9b2f0ee39b4efa93370234003a35d0" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 33377 } } peers { permanent_uuid: "200282175f6d4993a9937351cf946aeb" member_type: VOTER last_known_addr { host: "127.17.53.125" port: 43633 } } peers { permanent_uuid: "27e2b4963a95424090854f955bf1e1d2" member_type: VOTER last_known_addr { host: "127.17.53.124" port: 41889 } } }
I20260929 20:28:19.919950 7885 sys_catalog.cc:458] T 00000000000000000000000000000000 P 200282175f6d4993a9937351cf946aeb [sys.catalog]: This master's current role is: FOLLOWER
I20260929 20:28:19.921705 7842 sys_catalog.cc:455] T 00000000000000000000000000000000 P 27e2b4963a95424090854f955bf1e1d2 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3f9b2f0ee39b4efa93370234003a35d0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f9b2f0ee39b4efa93370234003a35d0" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 33377 } } peers { permanent_uuid: "200282175f6d4993a9937351cf946aeb" member_type: VOTER last_known_addr { host: "127.17.53.125" port: 43633 } } peers { permanent_uuid: "27e2b4963a95424090854f955bf1e1d2" member_type: VOTER last_known_addr { host: "127.17.53.124" port: 41889 } } }
I20260929 20:28:19.921957 7842 sys_catalog.cc:458] T 00000000000000000000000000000000 P 27e2b4963a95424090854f955bf1e1d2 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 20:28:19.922722 7735 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 200282175f6d4993a9937351cf946aeb [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:28:19.923636 7868 catalog_manager.cc:1347] Generated new cluster ID: ce0e2853196847988159b03ec411af49
I20260929 20:28:19.923743 7868 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:28:19.926620 7735 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 200282175f6d4993a9937351cf946aeb [term 1 FOLLOWER]: Refusing update from remote peer 3f9b2f0ee39b4efa93370234003a35d0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:28:19.922943 7736 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3f9b2f0ee39b4efa93370234003a35d0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "200282175f6d4993a9937351cf946aeb"
I20260929 20:28:19.927542 7856 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 3f9b2f0ee39b4efa93370234003a35d0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "200282175f6d4993a9937351cf946aeb" member_type: VOTER last_known_addr { host: "127.17.53.125" port: 43633 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:28:19.930078 7736 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 200282175f6d4993a9937351cf946aeb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f9b2f0ee39b4efa93370234003a35d0 in term 1.
I20260929 20:28:19.931005 7761 sys_catalog.cc:564] T 00000000000000000000000000000000 P 200282175f6d4993a9937351cf946aeb [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:28:19.951289 7868 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 20:28:19.953420 7868 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:28:19.966261 7859 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3f9b2f0ee39b4efa93370234003a35d0 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3f9b2f0ee39b4efa93370234003a35d0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f9b2f0ee39b4efa93370234003a35d0" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 33377 } } peers { permanent_uuid: "200282175f6d4993a9937351cf946aeb" member_type: VOTER last_known_addr { host: "127.17.53.125" port: 43633 } } peers { permanent_uuid: "27e2b4963a95424090854f955bf1e1d2" member_type: VOTER last_known_addr { host: "127.17.53.124" port: 41889 } } }
I20260929 20:28:19.966578 7859 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3f9b2f0ee39b4efa93370234003a35d0 [sys.catalog]: This master's current role is: LEADER
I20260929 20:28:19.966367 7885 sys_catalog.cc:455] T 00000000000000000000000000000000 P 200282175f6d4993a9937351cf946aeb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3f9b2f0ee39b4efa93370234003a35d0. Latest consensus state: current_term: 1 leader_uuid: "3f9b2f0ee39b4efa93370234003a35d0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f9b2f0ee39b4efa93370234003a35d0" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 33377 } } peers { permanent_uuid: "200282175f6d4993a9937351cf946aeb" member_type: VOTER last_known_addr { host: "127.17.53.125" port: 43633 } } peers { permanent_uuid: "27e2b4963a95424090854f955bf1e1d2" member_type: VOTER last_known_addr { host: "127.17.53.124" port: 41889 } } }
I20260929 20:28:19.966673 7885 sys_catalog.cc:458] T 00000000000000000000000000000000 P 200282175f6d4993a9937351cf946aeb [sys.catalog]: This master's current role is: FOLLOWER
I20260929 20:28:19.970898 7891 sys_catalog.cc:455] T 00000000000000000000000000000000 P 200282175f6d4993a9937351cf946aeb [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3f9b2f0ee39b4efa93370234003a35d0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3f9b2f0ee39b4efa93370234003a35d0" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 33377 } } peers { permanent_uuid: "200282175f6d4993a9937351cf946aeb" member_type: VOTER last_known_addr { host: "127.17.53.125" port: 43633 } } peers { permanent_uuid: "27e2b4963a95424090854f955bf1e1d2" member_type: VOTER last_known_addr { host: "127.17.53.124" port: 41889 } } }
I20260929 20:28:19.972000 7891 sys_catalog.cc:458] T 00000000000000000000000000000000 P 200282175f6d4993a9937351cf946aeb [sys.catalog]: This master's current role is: FOLLOWER
I20260929 20:28:19.988070 7868 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3f9b2f0ee39b4efa93370234003a35d0: Generated new TSK 0
I20260929 20:28:19.989159 7868 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 20:28:20.085871 7905 catalog_manager.cc:1259] Loaded cluster ID: ce0e2853196847988159b03ec411af49
I20260929 20:28:20.086002 7905 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 200282175f6d4993a9937351cf946aeb: loading cluster ID for follower catalog manager: success
I20260929 20:28:20.089656 7905 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 200282175f6d4993a9937351cf946aeb: acquiring CA information for follower catalog manager: success
I20260929 20:28:20.092118 7905 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 200282175f6d4993a9937351cf946aeb: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260929 20:28:20.130031 7839 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:28:20.102794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45136 (local address 127.17.53.126:33377)
0929 20:28:20.102986 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0929 20:28:20.102993 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:28:20.126499 (+ 23506us) server_negotiation.cc:407] Connection header received
0929 20:28:20.128404 (+ 1905us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:28:20.128411 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:28:20.128495 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:28:20.128566 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:28:20.129785 (+ 1219us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45136: BlockingRecv error: recv got EOF from 127.0.0.1:45136 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
W20260929 20:28:20.134677 7765 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:28:20.109045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57364 (local address 127.17.53.125:43633)
0929 20:28:20.109229 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0929 20:28:20.109241 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0929 20:28:20.131894 (+ 22653us) server_negotiation.cc:407] Connection header received
0929 20:28:20.133173 (+ 1279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:28:20.133180 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:28:20.133293 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:28:20.133367 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:28:20.134438 (+ 1071us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57364: BlockingRecv error: recv got EOF from 127.0.0.1:57364 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
W20260929 20:28:20.136719 7838 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:28:20.120635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60936 (local address 127.17.53.124:41889)
0929 20:28:20.120851 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0929 20:28:20.120858 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:28:20.132327 (+ 11469us) server_negotiation.cc:407] Connection header received
0929 20:28:20.135228 (+ 2901us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:28:20.135235 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:28:20.135316 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:28:20.135389 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:28:20.136459 (+ 1070us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60936: BlockingRecv error: recv got EOF from 127.0.0.1:60936 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
I20260929 20:28:20.184077 7765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:28:20.168045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57368 (local address 127.17.53.125:43633)
0929 20:28:20.168244 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0929 20:28:20.168250 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:28:20.175955 (+ 7705us) server_negotiation.cc:407] Connection header received
0929 20:28:20.176509 (+ 554us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:28:20.176516 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:28:20.176725 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:28:20.176885 (+ 160us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 20:28:20.177613 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:28:20.178811 (+ 1198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:28:20.180889 (+ 2078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:28:20.181560 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:28:20.181663 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:28:20.182453 (+ 790us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 20:28:20.183230 (+ 777us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 20:28:20.183349 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0929 20:28:20.183616 (+ 267us) server_negotiation.cc:299] Negotiation successful
0929 20:28:20.183713 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
W20260929 20:28:20.227401 7838 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:28:20.218059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60942 (local address 127.17.53.124:41889)
0929 20:28:20.218294 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0929 20:28:20.218301 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:28:20.225181 (+ 6880us) server_negotiation.cc:407] Connection header received
0929 20:28:20.225724 (+ 543us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:28:20.225731 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:28:20.225875 (+ 144us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:28:20.225955 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:28:20.227129 (+ 1174us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60942: BlockingRecv error: recv got EOF from 127.0.0.1:60942 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
I20260929 20:28:20.276049 7765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:28:20.261055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57376 (local address 127.17.53.125:43633)
0929 20:28:20.261258 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0929 20:28:20.261265 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:28:20.268807 (+ 7542us) server_negotiation.cc:407] Connection header received
0929 20:28:20.269370 (+ 563us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:28:20.269378 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:28:20.269474 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:28:20.269540 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 20:28:20.270260 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:28:20.271541 (+ 1281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:28:20.273722 (+ 2181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:28:20.274415 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:28:20.274507 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:28:20.275152 (+ 645us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 20:28:20.275355 (+ 203us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 20:28:20.275416 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0929 20:28:20.275642 (+ 226us) server_negotiation.cc:299] Negotiation successful
0929 20:28:20.275742 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
W20260929 20:28:20.317499 7838 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:28:20.308093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60944 (local address 127.17.53.124:41889)
0929 20:28:20.308336 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0929 20:28:20.308343 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:28:20.315554 (+ 7211us) server_negotiation.cc:407] Connection header received
0929 20:28:20.316028 (+ 474us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:28:20.316034 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:28:20.316129 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:28:20.316209 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:28:20.317269 (+ 1060us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60944: BlockingRecv error: recv got EOF from 127.0.0.1:60944 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
I20260929 20:28:20.373037 7765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:28:20.357549 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45540 (local address 127.17.53.125:43633)
0929 20:28:20.357767 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0929 20:28:20.357774 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:28:20.365358 (+ 7584us) server_negotiation.cc:407] Connection header received
0929 20:28:20.365916 (+ 558us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:28:20.365923 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:28:20.366019 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:28:20.366088 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 20:28:20.366903 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:28:20.368120 (+ 1217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:28:20.370492 (+ 2372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:28:20.371174 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:28:20.371281 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:28:20.371996 (+ 715us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 20:28:20.372181 (+ 185us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 20:28:20.372254 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0929 20:28:20.372549 (+ 295us) server_negotiation.cc:299] Negotiation successful
0929 20:28:20.372671 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
W20260929 20:28:20.397560 7838 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:28:20.393217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39028 (local address 127.17.53.124:41889)
0929 20:28:20.393409 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0929 20:28:20.393415 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:28:20.395466 (+ 2051us) server_negotiation.cc:407] Connection header received
0929 20:28:20.395962 (+ 496us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:28:20.395969 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:28:20.396073 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:28:20.396190 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:28:20.397303 (+ 1113us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39028: BlockingRecv error: recv got EOF from 127.0.0.1:39028 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
I20260929 20:28:20.416388 7765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:28:20.408993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45554 (local address 127.17.53.125:43633)
0929 20:28:20.409193 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0929 20:28:20.409200 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:28:20.409838 (+ 638us) server_negotiation.cc:407] Connection header received
0929 20:28:20.410078 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:28:20.410084 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:28:20.410199 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:28:20.410298 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 20:28:20.410886 (+ 588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:28:20.411991 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:28:20.414079 (+ 2088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:28:20.414773 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:28:20.414987 (+ 214us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:28:20.415507 (+ 520us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 20:28:20.415661 (+ 154us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 20:28:20.415761 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0929 20:28:20.415959 (+ 198us) server_negotiation.cc:299] Negotiation successful
0929 20:28:20.416067 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
W20260929 20:28:20.429356 7838 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:28:20.424066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39038 (local address 127.17.53.124:41889)
0929 20:28:20.424381 (+ 315us) server_negotiation.cc:206] Beginning negotiation
0929 20:28:20.424389 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 20:28:20.427607 (+ 3218us) server_negotiation.cc:407] Connection header received
0929 20:28:20.427834 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:28:20.427840 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:28:20.427952 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:28:20.428085 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:28:20.429105 (+ 1020us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39038: BlockingRecv error: recv got EOF from 127.0.0.1:39038 (error 108)
Metrics: {"server-negotiator.queue_time_us":171}
I20260929 20:28:20.450537 7765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:28:20.442739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45564 (local address 127.17.53.125:43633)
0929 20:28:20.442968 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0929 20:28:20.442976 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 20:28:20.443567 (+ 591us) server_negotiation.cc:407] Connection header received
0929 20:28:20.443794 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:28:20.443800 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:28:20.443905 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:28:20.444029 (+ 124us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 20:28:20.444572 (+ 543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:28:20.445810 (+ 1238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:28:20.447987 (+ 2177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:28:20.448836 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:28:20.448984 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:28:20.449541 (+ 557us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 20:28:20.449772 (+ 231us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 20:28:20.449873 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0929 20:28:20.450086 (+ 213us) server_negotiation.cc:299] Negotiation successful
0929 20:28:20.450186 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
W20260929 20:28:20.466223 7838 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:28:20.463437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39046 (local address 127.17.53.124:41889)
0929 20:28:20.463646 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0929 20:28:20.463653 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:28:20.464227 (+ 574us) server_negotiation.cc:407] Connection header received
0929 20:28:20.464617 (+ 390us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:28:20.464623 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:28:20.464739 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:28:20.464822 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:28:20.465955 (+ 1133us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39046: BlockingRecv error: recv got EOF from 127.0.0.1:39046 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
I20260929 20:28:20.497308 7765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:28:20.487485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45566 (local address 127.17.53.125:43633)
0929 20:28:20.487706 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0929 20:28:20.487712 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:28:20.488419 (+ 707us) server_negotiation.cc:407] Connection header received
0929 20:28:20.489029 (+ 610us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:28:20.489041 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:28:20.489164 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:28:20.489244 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 20:28:20.490093 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:28:20.491883 (+ 1790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:28:20.494325 (+ 2442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:28:20.495048 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:28:20.495192 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:28:20.495935 (+ 743us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 20:28:20.496160 (+ 225us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 20:28:20.496244 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0929 20:28:20.496524 (+ 280us) server_negotiation.cc:299] Negotiation successful
0929 20:28:20.496695 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
W20260929 20:28:20.507665 7838 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:28:20.505155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39050 (local address 127.17.53.124:41889)
0929 20:28:20.505388 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0929 20:28:20.505395 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:28:20.505926 (+ 531us) server_negotiation.cc:407] Connection header received
0929 20:28:20.506143 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:28:20.506149 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:28:20.506259 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:28:20.506370 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:28:20.507405 (+ 1035us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39050: BlockingRecv error: recv got EOF from 127.0.0.1:39050 (error 108)
Metrics: {"server-negotiator.queue_time_us":86}
I20260929 20:28:20.553464 7765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:28:20.546086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45578 (local address 127.17.53.125:43633)
0929 20:28:20.546298 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0929 20:28:20.546304 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:28:20.546931 (+ 627us) server_negotiation.cc:407] Connection header received
0929 20:28:20.547217 (+ 286us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:28:20.547223 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:28:20.547331 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:28:20.547438 (+ 107us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 20:28:20.547950 (+ 512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:28:20.549104 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:28:20.551229 (+ 2125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:28:20.551869 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:28:20.551988 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:28:20.552593 (+ 605us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 20:28:20.552727 (+ 134us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 20:28:20.552814 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0929 20:28:20.553017 (+ 203us) server_negotiation.cc:299] Negotiation successful
0929 20:28:20.553104 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
W20260929 20:28:20.562927 7838 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:28:20.560688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39060 (local address 127.17.53.124:41889)
0929 20:28:20.560891 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0929 20:28:20.560898 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:28:20.561187 (+ 289us) server_negotiation.cc:407] Connection header received
0929 20:28:20.561406 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:28:20.561411 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:28:20.561488 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:28:20.561573 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:28:20.562555 (+ 982us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39060: BlockingRecv error: recv got EOF from 127.0.0.1:39060 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
I20260929 20:28:20.636559 7765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:28:20.631237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45582 (local address 127.17.53.125:43633)
0929 20:28:20.631513 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0929 20:28:20.631519 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:28:20.631914 (+ 395us) server_negotiation.cc:407] Connection header received
0929 20:28:20.632164 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:28:20.632168 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:28:20.632253 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:28:20.632349 (+ 96us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 20:28:20.632811 (+ 462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:28:20.633523 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:28:20.634941 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:28:20.635360 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:28:20.635453 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:28:20.635848 (+ 395us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 20:28:20.635945 (+ 97us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 20:28:20.636009 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0929 20:28:20.636177 (+ 168us) server_negotiation.cc:299] Negotiation successful
0929 20:28:20.636251 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
W20260929 20:28:20.644177 7838 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:28:20.642095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39072 (local address 127.17.53.124:41889)
0929 20:28:20.642367 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0929 20:28:20.642375 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 20:28:20.642725 (+ 350us) server_negotiation.cc:407] Connection header received
0929 20:28:20.642970 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:28:20.642975 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:28:20.643060 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:28:20.643184 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:28:20.643997 (+ 813us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39072: BlockingRecv error: recv got EOF from 127.0.0.1:39072 (error 108)
Metrics: {"server-negotiator.queue_time_us":99}
I20260929 20:28:20.720266 7853 catalog_manager.cc:1259] Loaded cluster ID: ce0e2853196847988159b03ec411af49
I20260929 20:28:20.720377 7853 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 27e2b4963a95424090854f955bf1e1d2: loading cluster ID for follower catalog manager: success
I20260929 20:28:20.723031 7853 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 27e2b4963a95424090854f955bf1e1d2: acquiring CA information for follower catalog manager: success
I20260929 20:28:20.724860 7853 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 27e2b4963a95424090854f955bf1e1d2: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 20:28:20.781446 7765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:28:20.776056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45596 (local address 127.17.53.125:43633)
0929 20:28:20.776324 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0929 20:28:20.776334 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 20:28:20.776638 (+ 304us) server_negotiation.cc:407] Connection header received
0929 20:28:20.776902 (+ 264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:28:20.776907 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:28:20.776974 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:28:20.777080 (+ 106us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 20:28:20.777506 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:28:20.778257 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:28:20.779673 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:28:20.780114 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:28:20.780271 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:28:20.780704 (+ 433us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 20:28:20.780827 (+ 123us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 20:28:20.780937 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0929 20:28:20.781097 (+ 160us) server_negotiation.cc:299] Negotiation successful
0929 20:28:20.781178 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260929 20:28:20.793165 7838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:28:20.787532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39082 (local address 127.17.53.124:41889)
0929 20:28:20.787856 (+ 324us) server_negotiation.cc:206] Beginning negotiation
0929 20:28:20.787863 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:28:20.788085 (+ 222us) server_negotiation.cc:407] Connection header received
0929 20:28:20.788258 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:28:20.788262 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:28:20.788413 (+ 151us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:28:20.788513 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 20:28:20.788874 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:28:20.789527 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:28:20.790915 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:28:20.791322 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:28:20.791417 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:28:20.792003 (+ 586us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 20:28:20.792566 (+ 563us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 20:28:20.792692 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0929 20:28:20.792772 (+ 80us) server_negotiation.cc:299] Negotiation successful
0929 20:28:20.792837 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119}
I20260929 20:28:20.808928 7765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:28:20.800110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45608 (local address 127.17.53.125:43633)
0929 20:28:20.800373 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0929 20:28:20.800382 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 20:28:20.802839 (+ 2457us) server_negotiation.cc:407] Connection header received
0929 20:28:20.803001 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:28:20.803010 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:28:20.803132 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:28:20.803255 (+ 123us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 20:28:20.803850 (+ 595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:28:20.804827 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:28:20.806935 (+ 2108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:28:20.807379 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:28:20.807527 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:28:20.808126 (+ 599us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 20:28:20.808258 (+ 132us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 20:28:20.808362 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0929 20:28:20.808525 (+ 163us) server_negotiation.cc:299] Negotiation successful
0929 20:28:20.808609 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260929 20:28:20.811198 8009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:28:20.799926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47264 (local address 127.17.53.126:33377)
0929 20:28:20.801457 (+ 1531us) server_negotiation.cc:206] Beginning negotiation
0929 20:28:20.801467 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 20:28:20.802063 (+ 596us) server_negotiation.cc:407] Connection header received
0929 20:28:20.802358 (+ 295us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:28:20.802366 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:28:20.802620 (+ 254us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:28:20.802815 (+ 195us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 20:28:20.805034 (+ 2219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:28:20.806140 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:28:20.808630 (+ 2490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:28:20.809125 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:28:20.809238 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:28:20.809900 (+ 662us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 20:28:20.810437 (+ 537us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 20:28:20.810543 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0929 20:28:20.810745 (+ 202us) server_negotiation.cc:299] Negotiation successful
0929 20:28:20.810842 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":746,"thread_start_us":611,"threads_started":1}
I20260929 20:28:20.814291 7838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:28:20.801117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39094 (local address 127.17.53.124:41889)
0929 20:28:20.801341 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0929 20:28:20.801349 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 20:28:20.808081 (+ 6732us) server_negotiation.cc:407] Connection header received
0929 20:28:20.808508 (+ 427us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:28:20.808513 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:28:20.808601 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:28:20.808684 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 20:28:20.809520 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:28:20.810396 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:28:20.812223 (+ 1827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:28:20.812790 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:28:20.812925 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:28:20.813484 (+ 559us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 20:28:20.813623 (+ 139us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 20:28:20.813708 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0929 20:28:20.813933 (+ 225us) server_negotiation.cc:299] Negotiation successful
0929 20:28:20.814016 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260929 20:28:20.816257 17621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu pid 7620
I20260929 20:28:20.817914 7682 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:28:20.928965 17621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu with pid 7620
W20260929 20:28:20.949957 7710 connection.cc:535] server connection from 127.0.0.1:57324 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260929 20:28:20.950604 17621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu pid 7694
I20260929 20:28:20.951828 7756 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:28:21.062773 17621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu with pid 7694
I20260929 20:28:21.083244 17621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu pid 7768
I20260929 20:28:21.084353 7830 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:28:21.191568 17621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu with pid 7768
2026-09-29T20:28:21Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (4437 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790713339161492-17621-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 20:28:21 dist-test-slave-zbnc krb5kdc[8017](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 20:28:21 dist-test-slave-zbnc krb5kdc[8017](info): set up 2 sockets
Sep 29 20:28:21 dist-test-slave-zbnc krb5kdc[8017](info): commencing operation
krb5kdc: starting...
W20260929 20:28:22.340873 17621 mini_kdc.cc:121] Time spent starting KDC: real 1.108s user 0.000s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790713339161492-17621-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790713339161492-17621-0/krb5kdc/test-user.keytab.
Sep 29 20:28:22 dist-test-slave-zbnc krb5kdc[8017](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713702, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T20:28:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-29T20:28:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.17.53.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.53.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.53.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.126.keytab.
Entry for principal kudu/127.17.53.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.126.keytab.
I20260929 20:28:22.619223 17621 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
/tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790713339161492-17621-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790713339161492-17621-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790713339161492-17621-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790713339161492-17621-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.53.126:32843
--webserver_interface=127.17.53.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.126.keytab
--principal=kudu/127.17.53.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.53.84:37549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.53.126:32843,127.17.53.125:34869,127.17.53.124:32835
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790713339161492-17621-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790713339161492-17621-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790713339161492-17621-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 20:28:22.957091 8030 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 20:28:22.957484 8030 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:28:22.957625 8030 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:28:22.957775 8030 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:28:22.965476 8030 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:28:22.965572 8030 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:28:22.965678 8030 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:28:22.965812 8030 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:28:22.965909 8030 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:28:22.981285 8030 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.53.84:37549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790713339161492-17621-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790713339161492-17621-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.17.53.126:32843,127.17.53.125:34869,127.17.53.124:32835
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.126.keytab
--principal=kudu/127.17.53.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.53.126:32843
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790713339161492-17621-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.17.53.126
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790713339161492-17621-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:28:22.983299 8030 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:28:22.985247 8030 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:28:23.000550 8035 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:28:23.000576 8038 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:28:23.000514 8036 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:28:23.000994 8030 server_base.cc:1034] running on GCE node
I20260929 20:28:23.002318 8030 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:28:23.004319 8030 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:28:23.005578 8030 hybrid_clock.cc:648] HybridClock initialized: now 1790713703005487 us; error 60 us; skew 500 ppm
I20260929 20:28:23.005909 8030 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:28:23 dist-test-slave-zbnc krb5kdc[8017](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713703, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.53.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:28:23.010489 8030 init.cc:377] Logged in from keytab as kudu/127.17.53.126@KRBTEST.COM (short username kudu)
I20260929 20:28:23.012609 8030 webserver.cc:460] Webserver started at http://127.17.53.126:35929/ using document root <none> and password file <none>
I20260929 20:28:23.013625 8030 fs_manager.cc:359] Metadata directory not provided
I20260929 20:28:23.013770 8030 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:28:23.014201 8030 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:28:23.018953 8030 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790713339161492-17621-0/minicluster-data/master-0/data/instance:
uuid: "0b34a238284042f29227fcb824a834b2"
format_stamp: "Formatted at 2026-09-29 20:28:23 on dist-test-slave-zbnc"
server_key: "f0cce3852fef84628523546c19c6dedc"
server_key_iv: "3162051d93a65509ae84a921f56888f4"
server_key_version: "encryptionkey@0"
I20260929 20:28:23.020128 8030 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790713339161492-17621-0/minicluster-data/master-0/wal/instance:
uuid: "0b34a238284042f29227fcb824a834b2"
format_stamp: "Formatted at 2026-09-29 20:28:23 on dist-test-slave-zbnc"
server_key: "f0cce3852fef84628523546c19c6dedc"
server_key_iv: "3162051d93a65509ae84a921f56888f4"
server_key_version: "encryptionkey@0"
I20260929 20:28:23.027895 8030 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.001s
I20260929 20:28:23.031647 8045 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:28:23.033128 8030 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 20:28:23.033330 8030 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790713339161492-17621-0/minicluster-data/master-0/data,/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790713339161492-17621-0/minicluster-data/master-0/wal
uuid: "0b34a238284042f29227fcb824a834b2"
format_stamp: "Formatted at 2026-09-29 20:28:23 on dist-test-slave-zbnc"
server_key: "f0cce3852fef84628523546c19c6dedc"
server_key_iv: "3162051d93a65509ae84a921f56888f4"
server_key_version: "encryptionkey@0"
I20260929 20:28:23.033589 8030 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790713339161492-17621-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790713339161492-17621-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790713339161492-17621-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:28:23.050006 8030 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:28:23.056605 8030 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:28:23.056939 8030 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:28:23.078202 8030 rpc_server.cc:307] RPC server started. Bound to: 127.17.53.126:32843
I20260929 20:28:23.078213 8096 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.53.126:32843 every 8 connection(s)
I20260929 20:28:23.079706 8030 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790713339161492-17621-0/minicluster-data/master-0/data/info.pb
I20260929 20:28:23.083570 8097 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:28:23.087209 17621 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu as pid 8030
I20260929 20:28:23.087405 17621 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790713339161492-17621-0/minicluster-data/master-0/wal/instance
I20260929 20:28:23.087848 17621 external_mini_cluster.cc:1428] Setting key dae6c9af05c5ae48af097e4633ecf4f6
I20260929 20:28:23.090374 8097 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.53.126" port: 32843 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 20:28:23 dist-test-slave-zbnc krb5kdc[8017](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713703, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.53.126@KRBTEST.COM for kudu/127.17.53.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.17.53.125@KRBTEST.COM; defaulting to no policy
I20260929 20:28:23.115761 8101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:28:23.095008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46750 (local address 127.17.53.126:32843)
0929 20:28:23.095816 (+ 808us) server_negotiation.cc:206] Beginning negotiation
0929 20:28:23.095826 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 20:28:23.095890 (+ 64us) server_negotiation.cc:407] Connection header received
0929 20:28:23.097146 (+ 1256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:28:23.097157 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:28:23.097463 (+ 306us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:28:23.097689 (+ 226us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:28:23.099162 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:28:23.101049 (+ 1887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:28:23.102478 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:28:23.103482 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:28:23.103687 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:28:23.107209 (+ 3522us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:28:23.107222 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:28:23.107242 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:28:23.107287 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:28:23.111423 (+ 4136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:28:23.112175 (+ 752us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:28:23.112186 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:28:23.112190 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:28:23.112413 (+ 223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:28:23.112832 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:28:23.112837 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:28:23.112840 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:28:23.113565 (+ 725us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:28:23.113814 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0929 20:28:23.114333 (+ 519us) server_negotiation.cc:299] Negotiation successful
0929 20:28:23.114565 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":541,"thread_start_us":372,"threads_started":1}
I20260929 20:28:23.115803 8099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:28:23.093652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:32843 (local address 127.0.0.1:46750)
0929 20:28:23.094492 (+ 840us) negotiation.cc:107] Waiting for socket to connect
0929 20:28:23.094541 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0929 20:28:23.096727 (+ 2186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:28:23.097753 (+ 1026us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:28:23.097762 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:28:23.098539 (+ 777us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:28:23.098942 (+ 403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:28:23.098963 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:28:23.101213 (+ 2250us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:28:23.101216 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:28:23.102292 (+ 1076us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:28:23.102308 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:28:23.103702 (+ 1394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:28:23.103707 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:28:23.103862 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:28:23.104607 (+ 745us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:28:23.104635 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:28:23.106963 (+ 2328us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:28:23.111562 (+ 4599us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:28:23.111573 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:28:23.111583 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:28:23.112008 (+ 425us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:28:23.112536 (+ 528us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:28:23.112539 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:28:23.112541 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:28:23.112693 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:28:23.113735 (+ 1042us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:28:23.113743 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:28:23.113961 (+ 218us) client_negotiation.cc:769] Sending connection context
0929 20:28:23.114173 (+ 212us) client_negotiation.cc:240] Negotiation successful
0929 20:28:23.114724 (+ 551us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":567,"mutex_wait_us":141,"thread_start_us":370,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.53.125@KRBTEST.COM" created.
I20260929 20:28:23.119158 8097 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.53.125" port: 34869 } has no permanent_uuid. Determining permanent_uuid...
I20260929 20:28:23.120400 8099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:28:23.119883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.125:34869 (local address 127.0.0.1:52336)
0929 20:28:23.120101 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0929 20:28:23.120239 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.53.125:34869: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
W20260929 20:28:23.120733 8047 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.53.125:34869: connect: Connection refused (error 111)
I20260929 20:28:23.122427 8099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:28:23.121965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.125:34869 (local address 127.0.0.1:52352)
0929 20:28:23.122160 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0929 20:28:23.122254 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.53.125:34869: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260929 20:28:23.122632 8097 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.17.53.125:34869: Network error: Client connection negotiation failed: client connection to 127.17.53.125:34869: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.53.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.125.keytab.
Entry for principal kudu/127.17.53.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.125.keytab.
I20260929 20:28:23.150827 17621 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
/tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790713339161492-17621-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790713339161492-17621-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790713339161492-17621-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790713339161492-17621-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.53.125:34869
--webserver_interface=127.17.53.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.125.keytab
--principal=kudu/127.17.53.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.53.84:37549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.53.126:32843,127.17.53.125:34869,127.17.53.124:32835
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790713339161492-17621-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790713339161492-17621-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790713339161492-17621-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 20:28:23.171977 8097 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.53.125" port: 34869 } attempt: 1
I20260929 20:28:23.173273 8099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:28:23.172802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.125:34869 (local address 127.0.0.1:52364)
0929 20:28:23.173018 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0929 20:28:23.173144 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.53.125:34869: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260929 20:28:23.174299 8099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:28:23.173952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.125:34869 (local address 127.0.0.1:52366)
0929 20:28:23.174106 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0929 20:28:23.174181 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.53.125:34869: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260929 20:28:23.174484 8097 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.17.53.125:34869: Network error: Client connection negotiation failed: client connection to 127.17.53.125:34869: connect: Connection refused (error 111)
I20260929 20:28:23.242744 8097 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.53.125" port: 34869 } attempt: 2
I20260929 20:28:23.243969 8099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:28:23.243470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.125:34869 (local address 127.0.0.1:52368)
0929 20:28:23.243738 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0929 20:28:23.243832 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.53.125:34869: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260929 20:28:23.244961 8099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:28:23.244604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.125:34869 (local address 127.0.0.1:52384)
0929 20:28:23.244751 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0929 20:28:23.244841 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.53.125:34869: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260929 20:28:23.245134 8097 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.17.53.125:34869: Network error: Client connection negotiation failed: client connection to 127.17.53.125:34869: connect: Connection refused (error 111)
I20260929 20:28:23.336374 8097 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.53.125" port: 34869 } attempt: 3
I20260929 20:28:23.337568 8099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:28:23.337120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.125:34869 (local address 127.0.0.1:52388)
0929 20:28:23.337306 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0929 20:28:23.337435 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.53.125:34869: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260929 20:28:23.338608 8099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:28:23.338246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.125:34869 (local address 127.0.0.1:52394)
0929 20:28:23.338411 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0929 20:28:23.338477 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.53.125:34869: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260929 20:28:23.338780 8097 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.17.53.125:34869: Network error: Client connection negotiation failed: client connection to 127.17.53.125:34869: connect: Connection refused (error 111)
I20260929 20:28:23.482120 8097 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.53.125" port: 34869 } attempt: 4
I20260929 20:28:23.483444 8099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:28:23.482898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.125:34869 (local address 127.0.0.1:52402)
0929 20:28:23.483204 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0929 20:28:23.483292 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.53.125:34869: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":162}
I20260929 20:28:23.484705 8099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:28:23.484141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.125:34869 (local address 127.0.0.1:52412)
0929 20:28:23.484397 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0929 20:28:23.484511 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.53.125:34869: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
W20260929 20:28:23.485163 8097 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.17.53.125:34869: Network error: Client connection negotiation failed: client connection to 127.17.53.125:34869: connect: Connection refused (error 111)
W20260929 20:28:23.492271 8104 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 20:28:23.492650 8104 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:28:23.492764 8104 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:28:23.492903 8104 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260929 20:28:23.500451 8104 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:28:23.500535 8104 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:28:23.500655 8104 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:28:23.500743 8104 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:28:23.500851 8104 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:28:23.510984 8104 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.53.84:37549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790713339161492-17621-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790713339161492-17621-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.17.53.126:32843,127.17.53.125:34869,127.17.53.124:32835
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.125.keytab
--principal=kudu/127.17.53.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.53.125:34869
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790713339161492-17621-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.17.53.125
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790713339161492-17621-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:28:23.512382 8104 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:28:23.513710 8104 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260929 20:28:23.526036 8109 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:28:23.526013 8110 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:28:23.526068 8112 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:28:23.526731 8104 server_base.cc:1034] running on GCE node
I20260929 20:28:23.527571 8104 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:28:23.529032 8104 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:28:23.530299 8104 hybrid_clock.cc:648] HybridClock initialized: now 1790713703530216 us; error 57 us; skew 500 ppm
I20260929 20:28:23.530646 8104 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:28:23 dist-test-slave-zbnc krb5kdc[8017](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713703, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.53.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:28:23.534112 8104 init.cc:377] Logged in from keytab as kudu/127.17.53.125@KRBTEST.COM (short username kudu)
I20260929 20:28:23.535673 8104 webserver.cc:460] Webserver started at http://127.17.53.125:33593/ using document root <none> and password file <none>
I20260929 20:28:23.536343 8104 fs_manager.cc:359] Metadata directory not provided
I20260929 20:28:23.536432 8104 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:28:23.536755 8104 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:28:23.540017 8104 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790713339161492-17621-0/minicluster-data/master-1/data/instance:
uuid: "842df621ae7c44db813a0ed42fe76b0f"
format_stamp: "Formatted at 2026-09-29 20:28:23 on dist-test-slave-zbnc"
server_key: "252e06a2915ec3a7fb572d047773a328"
server_key_iv: "c7c17bf551233cc3b4a8da8ce95c3e4f"
server_key_version: "encryptionkey@0"
I20260929 20:28:23.540782 8104 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790713339161492-17621-0/minicluster-data/master-1/wal/instance:
uuid: "842df621ae7c44db813a0ed42fe76b0f"
format_stamp: "Formatted at 2026-09-29 20:28:23 on dist-test-slave-zbnc"
server_key: "252e06a2915ec3a7fb572d047773a328"
server_key_iv: "c7c17bf551233cc3b4a8da8ce95c3e4f"
server_key_version: "encryptionkey@0"
I20260929 20:28:23.546034 8104 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 20:28:23.549515 8119 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:28:23.550844 8104 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 20:28:23.551075 8104 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790713339161492-17621-0/minicluster-data/master-1/data,/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790713339161492-17621-0/minicluster-data/master-1/wal
uuid: "842df621ae7c44db813a0ed42fe76b0f"
format_stamp: "Formatted at 2026-09-29 20:28:23 on dist-test-slave-zbnc"
server_key: "252e06a2915ec3a7fb572d047773a328"
server_key_iv: "c7c17bf551233cc3b4a8da8ce95c3e4f"
server_key_version: "encryptionkey@0"
I20260929 20:28:23.551256 8104 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790713339161492-17621-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790713339161492-17621-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790713339161492-17621-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:28:23.570631 8104 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:28:23.575169 8104 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:28:23.575398 8104 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:28:23.593750 8104 rpc_server.cc:307] RPC server started. Bound to: 127.17.53.125:34869
I20260929 20:28:23.593787 8170 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.53.125:34869 every 8 connection(s)
I20260929 20:28:23.595217 8104 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790713339161492-17621-0/minicluster-data/master-1/data/info.pb
I20260929 20:28:23.598169 17621 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu as pid 8104
I20260929 20:28:23.598371 17621 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790713339161492-17621-0/minicluster-data/master-1/wal/instance
I20260929 20:28:23.598965 17621 external_mini_cluster.cc:1428] Setting key 0f042c88bb74e98dd17d072e5d598902
I20260929 20:28:23.599664 8171 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:28:23.609813 8171 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.53.126" port: 32843 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 20:28:23 dist-test-slave-zbnc krb5kdc[8017](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713703, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.53.125@KRBTEST.COM for kudu/127.17.53.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.17.53.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.17.53.124@KRBTEST.COM" created.
I20260929 20:28:23.631264 8101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:28:23.612886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46758 (local address 127.17.53.126:32843)
0929 20:28:23.613077 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0929 20:28:23.613081 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:28:23.614460 (+ 1379us) server_negotiation.cc:407] Connection header received
0929 20:28:23.616443 (+ 1983us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:28:23.616449 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:28:23.616547 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:28:23.616670 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:28:23.618204 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:28:23.618951 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:28:23.620464 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:28:23.620886 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:28:23.621034 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:28:23.624631 (+ 3597us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:28:23.624636 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:28:23.624639 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:28:23.624672 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:28:23.627613 (+ 2941us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:28:23.628457 (+ 844us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:28:23.628462 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:28:23.628466 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:28:23.628596 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:28:23.629095 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:28:23.629102 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:28:23.629106 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:28:23.629504 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:28:23.629671 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0929 20:28:23.630467 (+ 796us) server_negotiation.cc:299] Negotiation successful
0929 20:28:23.630737 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260929 20:28:23.632128 8174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:28:23.613483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:32843 (local address 127.0.0.1:46758)
0929 20:28:23.614285 (+ 802us) negotiation.cc:107] Waiting for socket to connect
0929 20:28:23.614322 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0929 20:28:23.616153 (+ 1831us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:28:23.616921 (+ 768us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:28:23.616930 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:28:23.617464 (+ 534us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:28:23.618026 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:28:23.618042 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:28:23.619114 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:28:23.619119 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:28:23.620304 (+ 1185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:28:23.620316 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:28:23.621124 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:28:23.621132 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:28:23.621389 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:28:23.622115 (+ 726us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:28:23.622137 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:28:23.624399 (+ 2262us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:28:23.627783 (+ 3384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:28:23.627793 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:28:23.627811 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:28:23.628319 (+ 508us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:28:23.628735 (+ 416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:28:23.628747 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:28:23.628751 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:28:23.628965 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:28:23.629681 (+ 716us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:28:23.629688 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:28:23.630082 (+ 394us) client_negotiation.cc:769] Sending connection context
0929 20:28:23.630341 (+ 259us) client_negotiation.cc:240] Negotiation successful
0929 20:28:23.630741 (+ 400us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":560,"thread_start_us":327,"threads_started":1}
I20260929 20:28:23.633994 8171 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.53.125" port: 34869 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 20:28:23 dist-test-slave-zbnc krb5kdc[8017](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713703, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.53.125@KRBTEST.COM for kudu/127.17.53.125@KRBTEST.COM
I20260929 20:28:23.651196 8174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:28:23.634919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.125:34869 (local address 127.0.0.1:52424)
0929 20:28:23.635110 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0929 20:28:23.635133 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0929 20:28:23.635238 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:28:23.636388 (+ 1150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:28:23.636393 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:28:23.636965 (+ 572us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:28:23.637101 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:28:23.637110 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:28:23.638766 (+ 1656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:28:23.638771 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:28:23.640262 (+ 1491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:28:23.640306 (+ 44us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:28:23.641184 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:28:23.641189 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:28:23.641462 (+ 273us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:28:23.642129 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:28:23.642147 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:28:23.644341 (+ 2194us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:28:23.648702 (+ 4361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:28:23.648707 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:28:23.648709 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:28:23.649081 (+ 372us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:28:23.649518 (+ 437us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:28:23.649522 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:28:23.649524 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:28:23.649659 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:28:23.650422 (+ 763us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:28:23.650425 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:28:23.650536 (+ 111us) client_negotiation.cc:769] Sending connection context
0929 20:28:23.650636 (+ 100us) client_negotiation.cc:240] Negotiation successful
0929 20:28:23.650848 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55,"mutex_wait_us":60}
I20260929 20:28:23.651713 8175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:28:23.635205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52424 (local address 127.17.53.125:34869)
0929 20:28:23.635962 (+ 757us) server_negotiation.cc:206] Beginning negotiation
0929 20:28:23.635968 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:28:23.635991 (+ 23us) server_negotiation.cc:407] Connection header received
0929 20:28:23.636036 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:28:23.636042 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:28:23.636223 (+ 181us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:28:23.636410 (+ 187us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:28:23.637287 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:28:23.638605 (+ 1318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:28:23.640471 (+ 1866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:28:23.641021 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:28:23.641174 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:28:23.644590 (+ 3416us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:28:23.644622 (+ 32us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:28:23.644644 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:28:23.644686 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:28:23.648553 (+ 3867us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:28:23.649235 (+ 682us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:28:23.649247 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:28:23.649253 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:28:23.649406 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:28:23.649787 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:28:23.649795 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:28:23.649800 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:28:23.650300 (+ 500us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:28:23.650498 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0929 20:28:23.650895 (+ 397us) server_negotiation.cc:299] Negotiation successful
0929 20:28:23.651196 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":589,"thread_start_us":453,"threads_started":1}
I20260929 20:28:23.654290 8171 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.53.124" port: 32835 } has no permanent_uuid. Determining permanent_uuid...
I20260929 20:28:23.655488 8174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:28:23.655011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.124:32835 (local address 127.0.0.1:33860)
0929 20:28:23.655191 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0929 20:28:23.655343 (+ 152us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.53.124:32835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260929 20:28:23.655797 8121 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.17.53.124:32835: connect: Connection refused (error 111)
I20260929 20:28:23.657389 8174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:28:23.657087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.124:32835 (local address 127.0.0.1:33864)
0929 20:28:23.657222 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0929 20:28:23.657297 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.53.124:32835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260929 20:28:23.657639 8171 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.17.53.124:32835: Network error: Client connection negotiation failed: client connection to 127.17.53.124:32835: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.17.53.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.124.keytab.
Entry for principal kudu/127.17.53.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.124.keytab.
I20260929 20:28:23.667666 17621 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
/tmp/dist-test-taskfEhy_t/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790713339161492-17621-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790713339161492-17621-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790713339161492-17621-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790713339161492-17621-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.53.124:32835
--webserver_interface=127.17.53.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.124.keytab
--principal=kudu/127.17.53.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.17.53.84:37549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.17.53.126:32843,127.17.53.125:34869,127.17.53.124:32835
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790713339161492-17621-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790713339161492-17621-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790713339161492-17621-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 20:28:23.706986 8171 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.53.124" port: 32835 } attempt: 1
I20260929 20:28:23.708148 8174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:28:23.707713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.124:32835 (local address 127.0.0.1:33876)
0929 20:28:23.707901 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0929 20:28:23.708003 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.53.124:32835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260929 20:28:23.709142 8174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:28:23.708787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.124:32835 (local address 127.0.0.1:33882)
0929 20:28:23.708959 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0929 20:28:23.709039 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.53.124:32835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260929 20:28:23.709374 8171 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.17.53.124:32835: Network error: Client connection negotiation failed: client connection to 127.17.53.124:32835: connect: Connection refused (error 111)
I20260929 20:28:23.777591 8171 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.53.124" port: 32835 } attempt: 2
I20260929 20:28:23.778776 8174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:28:23.778326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.124:32835 (local address 127.0.0.1:33884)
0929 20:28:23.778520 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0929 20:28:23.778620 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.53.124:32835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260929 20:28:23.779806 8174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:28:23.779474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.124:32835 (local address 127.0.0.1:33900)
0929 20:28:23.779628 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0929 20:28:23.779694 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.53.124:32835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260929 20:28:23.780011 8171 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.17.53.124:32835: Network error: Client connection negotiation failed: client connection to 127.17.53.124:32835: connect: Connection refused (error 111)
I20260929 20:28:23.784397 8097 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.53.125" port: 34869 } attempt: 5
Sep 29 20:28:23 dist-test-slave-zbnc krb5kdc[8017](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713703, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.53.126@KRBTEST.COM for kudu/127.17.53.125@KRBTEST.COM
I20260929 20:28:23.798748 8099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:28:23.784995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.125:34869 (local address 127.0.0.1:52434)
0929 20:28:23.785209 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0929 20:28:23.785230 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0929 20:28:23.785362 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:28:23.785694 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:28:23.785699 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:28:23.786211 (+ 512us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:28:23.786477 (+ 266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:28:23.786494 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:28:23.787562 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:28:23.787567 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:28:23.788926 (+ 1359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:28:23.788943 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:28:23.790046 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:28:23.790051 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:28:23.790278 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:28:23.790932 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:28:23.790955 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:28:23.793276 (+ 2321us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:28:23.796315 (+ 3039us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:28:23.796322 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:28:23.796327 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:28:23.796707 (+ 380us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:28:23.797084 (+ 377us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:28:23.797088 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:28:23.797092 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:28:23.797234 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:28:23.797766 (+ 532us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:28:23.797770 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:28:23.797932 (+ 162us) client_negotiation.cc:769] Sending connection context
0929 20:28:23.798080 (+ 148us) client_negotiation.cc:240] Negotiation successful
0929 20:28:23.798384 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":73}
I20260929 20:28:23.798761 8175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:28:23.785225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52434 (local address 127.17.53.125:34869)
0929 20:28:23.785388 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0929 20:28:23.785393 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:28:23.785433 (+ 40us) server_negotiation.cc:407] Connection header received
0929 20:28:23.785485 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:28:23.785489 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:28:23.785549 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:28:23.785621 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:28:23.786645 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:28:23.787415 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:28:23.789149 (+ 1734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:28:23.789881 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:28:23.789986 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:28:23.793470 (+ 3484us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:28:23.793476 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:28:23.793479 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:28:23.793511 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:28:23.796164 (+ 2653us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:28:23.796831 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:28:23.796838 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:28:23.796842 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:28:23.796962 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:28:23.797351 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:28:23.797358 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:28:23.797362 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:28:23.797640 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:28:23.797809 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0929 20:28:23.798190 (+ 381us) server_negotiation.cc:299] Negotiation successful
0929 20:28:23.798402 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 20:28:23.799880 8097 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.53.124" port: 32835 } has no permanent_uuid. Determining permanent_uuid...
I20260929 20:28:23.800864 8099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:28:23.800502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.124:32835 (local address 127.0.0.1:33910)
0929 20:28:23.800660 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0929 20:28:23.800734 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.53.124:32835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260929 20:28:23.801741 8099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:28:23.801422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.124:32835 (local address 127.0.0.1:33912)
0929 20:28:23.801566 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0929 20:28:23.801652 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.53.124:32835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260929 20:28:23.801964 8097 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.17.53.124:32835: Network error: Client connection negotiation failed: client connection to 127.17.53.124:32835: connect: Connection refused (error 111)
I20260929 20:28:23.853219 8097 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.53.124" port: 32835 } attempt: 1
I20260929 20:28:23.854496 8099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:28:23.853931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.124:32835 (local address 127.0.0.1:33922)
0929 20:28:23.854168 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0929 20:28:23.854286 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.53.124:32835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260929 20:28:23.855564 8099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:28:23.855207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.124:32835 (local address 127.0.0.1:33926)
0929 20:28:23.855347 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0929 20:28:23.855437 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.53.124:32835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260929 20:28:23.855904 8097 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.17.53.124:32835: Network error: Client connection negotiation failed: client connection to 127.17.53.124:32835: connect: Connection refused (error 111)
I20260929 20:28:23.871244 8171 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.53.124" port: 32835 } attempt: 3
I20260929 20:28:23.872498 8174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:28:23.872020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.124:32835 (local address 127.0.0.1:33940)
0929 20:28:23.872229 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0929 20:28:23.872345 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.53.124:32835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20260929 20:28:23.873720 8174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:28:23.873315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.124:32835 (local address 127.0.0.1:33942)
0929 20:28:23.873547 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0929 20:28:23.873618 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.53.124:32835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153}
W20260929 20:28:23.873999 8171 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.17.53.124:32835: Network error: Client connection negotiation failed: client connection to 127.17.53.124:32835: connect: Connection refused (error 111)
I20260929 20:28:23.924137 8097 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.53.124" port: 32835 } attempt: 2
I20260929 20:28:23.925355 8099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:28:23.924858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.124:32835 (local address 127.0.0.1:33944)
0929 20:28:23.925107 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0929 20:28:23.925197 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.53.124:32835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260929 20:28:23.926394 8099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:28:23.926035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.124:32835 (local address 127.0.0.1:33946)
0929 20:28:23.926200 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0929 20:28:23.926296 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.53.124:32835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260929 20:28:23.926614 8097 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.17.53.124:32835: Network error: Client connection negotiation failed: client connection to 127.17.53.124:32835: connect: Connection refused (error 111)
W20260929 20:28:24.012374 8178 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260929 20:28:24.012810 8178 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 20:28:24.012941 8178 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 20:28:24.013077 8178 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260929 20:28:24.017223 8171 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.53.124" port: 32835 } attempt: 4
I20260929 20:28:24.018396 8174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:28:24.017928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.124:32835 (local address 127.0.0.1:33950)
0929 20:28:24.018153 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0929 20:28:24.018246 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.53.124:32835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260929 20:28:24.019358 8174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:28:24.019041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.124:32835 (local address 127.0.0.1:33956)
0929 20:28:24.019186 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0929 20:28:24.019257 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.53.124:32835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260929 20:28:24.019563 8171 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.17.53.124:32835: Network error: Client connection negotiation failed: client connection to 127.17.53.124:32835: connect: Connection refused (error 111)
W20260929 20:28:24.020911 8178 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 20:28:24.021045 8178 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260929 20:28:24.021160 8178 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260929 20:28:24.021245 8178 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 20:28:24.021353 8178 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260929 20:28:24.031606 8178 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.17.53.84:37549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790713339161492-17621-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790713339161492-17621-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.17.53.126:32843,127.17.53.125:34869,127.17.53.124:32835
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790713339161492-17621-0/krb5kdc/kudu_127.17.53.124.keytab
--principal=kudu/127.17.53.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.17.53.124:32835
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790713339161492-17621-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.17.53.124
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790713339161492-17621-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 29 Sep 2026 18:53:27 UTC on ccycloud.quasar-ahfegm.root.comops.site
build id 4680914
ASAN enabled
I20260929 20:28:24.032812 8097 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.53.124" port: 32835 } attempt: 3
I20260929 20:28:24.033108 8178 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 20:28:24.033878 8099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:28:24.033419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.124:32835 (local address 127.0.0.1:33960)
0929 20:28:24.033647 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0929 20:28:24.033729 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.53.124:32835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260929 20:28:24.034579 8178 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260929 20:28:24.034919 8099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:28:24.034501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.124:32835 (local address 127.0.0.1:33966)
0929 20:28:24.034651 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0929 20:28:24.034760 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.17.53.124:32835: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260929 20:28:24.035130 8097 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.17.53.124:32835: Network error: Client connection negotiation failed: client connection to 127.17.53.124:32835: connect: Connection refused (error 111)
W20260929 20:28:24.046706 8184 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:28:24.046787 8186 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 20:28:24.046756 8183 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 20:28:24.047658 8178 server_base.cc:1034] running on GCE node
I20260929 20:28:24.048384 8178 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 20:28:24.049769 8178 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 20:28:24.050980 8178 hybrid_clock.cc:648] HybridClock initialized: now 1790713704050907 us; error 47 us; skew 500 ppm
I20260929 20:28:24.051365 8178 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 20:28:24 dist-test-slave-zbnc krb5kdc[8017](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713704, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.53.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 20:28:24.054489 8178 init.cc:377] Logged in from keytab as kudu/127.17.53.124@KRBTEST.COM (short username kudu)
I20260929 20:28:24.056197 8178 webserver.cc:460] Webserver started at http://127.17.53.124:43913/ using document root <none> and password file <none>
I20260929 20:28:24.056924 8178 fs_manager.cc:359] Metadata directory not provided
I20260929 20:28:24.057065 8178 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 20:28:24.057396 8178 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 20:28:24.060432 8178 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790713339161492-17621-0/minicluster-data/master-2/data/instance:
uuid: "9faeea581fc04e18a73c4146a01ddb8a"
format_stamp: "Formatted at 2026-09-29 20:28:24 on dist-test-slave-zbnc"
server_key: "f3df1454005ebbf4ade9e0ca1898f9c8"
server_key_iv: "6381ab0224db4a3380f34a38489b68ba"
server_key_version: "encryptionkey@0"
I20260929 20:28:24.061230 8178 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790713339161492-17621-0/minicluster-data/master-2/wal/instance:
uuid: "9faeea581fc04e18a73c4146a01ddb8a"
format_stamp: "Formatted at 2026-09-29 20:28:24 on dist-test-slave-zbnc"
server_key: "f3df1454005ebbf4ade9e0ca1898f9c8"
server_key_iv: "6381ab0224db4a3380f34a38489b68ba"
server_key_version: "encryptionkey@0"
I20260929 20:28:24.066493 8178 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260929 20:28:24.069978 8193 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 20:28:24.071364 8178 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 20:28:24.071551 8178 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790713339161492-17621-0/minicluster-data/master-2/data,/tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790713339161492-17621-0/minicluster-data/master-2/wal
uuid: "9faeea581fc04e18a73c4146a01ddb8a"
format_stamp: "Formatted at 2026-09-29 20:28:24 on dist-test-slave-zbnc"
server_key: "f3df1454005ebbf4ade9e0ca1898f9c8"
server_key_iv: "6381ab0224db4a3380f34a38489b68ba"
server_key_version: "encryptionkey@0"
I20260929 20:28:24.071777 8178 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790713339161492-17621-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790713339161492-17621-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790713339161492-17621-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 20:28:24.087714 8178 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260929 20:28:24.092406 8178 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 20:28:24.092722 8178 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 20:28:24.112418 8178 rpc_server.cc:307] RPC server started. Bound to: 127.17.53.124:32835
I20260929 20:28:24.112442 8244 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.17.53.124:32835 every 8 connection(s)
I20260929 20:28:24.113838 8178 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790713339161492-17621-0/minicluster-data/master-2/data/info.pb
I20260929 20:28:24.115082 17621 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu as pid 8178
I20260929 20:28:24.115291 17621 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfEhy_t/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790713339161492-17621-0/minicluster-data/master-2/wal/instance
I20260929 20:28:24.115876 17621 external_mini_cluster.cc:1428] Setting key d9f53e7e2a7491de87c3cae032b2d3e2
I20260929 20:28:24.118789 8245 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 20:28:24.126317 8245 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.53.126" port: 32843 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 20:28:24 dist-test-slave-zbnc krb5kdc[8017](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713702, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.53.124@KRBTEST.COM
Sep 29 20:28:24 dist-test-slave-zbnc krb5kdc[8017](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713704, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.53.124@KRBTEST.COM for kudu/127.17.53.126@KRBTEST.COM
I20260929 20:28:24.140717 8248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:28:24.117874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33974 (local address 127.17.53.124:32835)
0929 20:28:24.118643 (+ 769us) server_negotiation.cc:206] Beginning negotiation
0929 20:28:24.118656 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0929 20:28:24.118696 (+ 40us) server_negotiation.cc:407] Connection header received
0929 20:28:24.120385 (+ 1689us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:28:24.120407 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:28:24.120818 (+ 411us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:28:24.121283 (+ 465us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:28:24.122724 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:28:24.124163 (+ 1439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:28:24.125526 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:28:24.126198 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:28:24.126413 (+ 215us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:28:24.131491 (+ 5078us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:28:24.131503 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:28:24.131519 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:28:24.131572 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:28:24.135539 (+ 3967us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:28:24.136447 (+ 908us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:28:24.136454 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:28:24.136461 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:28:24.136853 (+ 392us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:28:24.137415 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:28:24.137422 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:28:24.137425 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:28:24.138012 (+ 587us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:28:24.138338 (+ 326us) server_negotiation.cc:1035] Waiting for connection context
0929 20:28:24.138926 (+ 588us) server_negotiation.cc:299] Negotiation successful
0929 20:28:24.139477 (+ 551us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":555,"spinlock_wait_cycles":20224,"thread_start_us":368,"threads_started":1}
I20260929 20:28:24.146984 8249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:28:24.129142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.126:32843 (local address 127.0.0.1:46762)
0929 20:28:24.129960 (+ 818us) negotiation.cc:107] Waiting for socket to connect
0929 20:28:24.129996 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0929 20:28:24.130211 (+ 215us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:28:24.130712 (+ 501us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:28:24.130719 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:28:24.131554 (+ 835us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:28:24.131983 (+ 429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:28:24.131998 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:28:24.133354 (+ 1356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:28:24.133361 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:28:24.135224 (+ 1863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:28:24.135242 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:28:24.136186 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:28:24.136191 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:28:24.136397 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:28:24.137225 (+ 828us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:28:24.137251 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:28:24.139926 (+ 2675us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:28:24.143899 (+ 3973us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:28:24.143910 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:28:24.143917 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:28:24.144506 (+ 589us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:28:24.144950 (+ 444us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:28:24.144956 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:28:24.144960 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:28:24.145098 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:28:24.145796 (+ 698us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:28:24.145808 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:28:24.146038 (+ 230us) client_negotiation.cc:769] Sending connection context
0929 20:28:24.146254 (+ 216us) client_negotiation.cc:240] Negotiation successful
0929 20:28:24.146616 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":609,"thread_start_us":337,"threads_started":1}
I20260929 20:28:24.147183 8101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:28:24.129364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46762 (local address 127.17.53.126:32843)
0929 20:28:24.129610 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0929 20:28:24.129615 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:28:24.130122 (+ 507us) server_negotiation.cc:407] Connection header received
0929 20:28:24.130382 (+ 260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:28:24.130388 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:28:24.130506 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:28:24.130644 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:28:24.132227 (+ 1583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:28:24.133167 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:28:24.135383 (+ 2216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:28:24.135982 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:28:24.136125 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:28:24.140160 (+ 4035us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:28:24.140165 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:28:24.140169 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:28:24.140206 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:28:24.143724 (+ 3518us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:28:24.144641 (+ 917us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:28:24.144647 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:28:24.144655 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:28:24.144821 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:28:24.145264 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:28:24.145275 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:28:24.145280 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:28:24.145632 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:28:24.145765 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0929 20:28:24.146405 (+ 640us) server_negotiation.cc:299] Negotiation successful
0929 20:28:24.146747 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260929 20:28:24.148375 8245 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.53.125" port: 34869 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 20:28:24 dist-test-slave-zbnc krb5kdc[8017](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713702, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.53.126@KRBTEST.COM
Sep 29 20:28:24 dist-test-slave-zbnc krb5kdc[8017](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713704, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.53.124@KRBTEST.COM for kudu/127.17.53.125@KRBTEST.COM
I20260929 20:28:24.160980 8250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:28:24.144252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46768 (local address 127.17.53.126:32843)
0929 20:28:24.144920 (+ 668us) server_negotiation.cc:206] Beginning negotiation
0929 20:28:24.144927 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:28:24.144948 (+ 21us) server_negotiation.cc:407] Connection header received
0929 20:28:24.144994 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:28:24.144997 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:28:24.145052 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:28:24.145165 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:28:24.146613 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:28:24.147898 (+ 1285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:28:24.149175 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:28:24.149647 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:28:24.149744 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:28:24.154924 (+ 5180us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:28:24.154931 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:28:24.154939 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:28:24.154976 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:28:24.158008 (+ 3032us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:28:24.158848 (+ 840us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:28:24.158852 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:28:24.158896 (+ 44us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:28:24.159018 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:28:24.159425 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:28:24.159428 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:28:24.159430 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:28:24.159699 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:28:24.159893 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0929 20:28:24.160349 (+ 456us) server_negotiation.cc:299] Negotiation successful
0929 20:28:24.160567 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":540,"thread_start_us":337,"threads_started":1}
I20260929 20:28:24.162838 8249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:28:24.149164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.125:34869 (local address 127.0.0.1:52440)
0929 20:28:24.149347 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0929 20:28:24.149366 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0929 20:28:24.149461 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:28:24.149861 (+ 400us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:28:24.149867 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:28:24.150225 (+ 358us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:28:24.150422 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:28:24.150435 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:28:24.151986 (+ 1551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:28:24.151990 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:28:24.152867 (+ 877us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:28:24.152879 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:28:24.153697 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:28:24.153703 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:28:24.153937 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:28:24.154618 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:28:24.154641 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:28:24.156727 (+ 2086us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:28:24.160268 (+ 3541us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:28:24.160274 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:28:24.160277 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:28:24.160752 (+ 475us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:28:24.161150 (+ 398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:28:24.161156 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:28:24.161159 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:28:24.161303 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:28:24.161932 (+ 629us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:28:24.161935 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:28:24.162146 (+ 211us) client_negotiation.cc:769] Sending connection context
0929 20:28:24.162272 (+ 126us) client_negotiation.cc:240] Negotiation successful
0929 20:28:24.162466 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":66}
I20260929 20:28:24.163265 8175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:28:24.149178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52440 (local address 127.17.53.125:34869)
0929 20:28:24.149392 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0929 20:28:24.149400 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 20:28:24.149418 (+ 18us) server_negotiation.cc:407] Connection header received
0929 20:28:24.149601 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:28:24.149606 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:28:24.149678 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:28:24.149809 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:28:24.150608 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:28:24.151824 (+ 1216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:28:24.153030 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:28:24.153515 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:28:24.153669 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:28:24.156935 (+ 3266us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:28:24.156941 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:28:24.156944 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:28:24.156975 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:28:24.160113 (+ 3138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:28:24.160874 (+ 761us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:28:24.160879 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:28:24.160881 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:28:24.161026 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:28:24.161417 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:28:24.161423 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:28:24.161427 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:28:24.161799 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:28:24.161966 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0929 20:28:24.162421 (+ 455us) server_negotiation.cc:299] Negotiation successful
0929 20:28:24.162615 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102}
I20260929 20:28:24.164481 8245 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.17.53.124" port: 32835 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 20:28:24 dist-test-slave-zbnc krb5kdc[8017](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713704, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.53.124@KRBTEST.COM for kudu/127.17.53.124@KRBTEST.COM
Sep 29 20:28:24 dist-test-slave-zbnc krb5kdc[8017](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713702, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.17.53.125@KRBTEST.COM
I20260929 20:28:24.179258 8175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:28:24.162926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52448 (local address 127.17.53.125:34869)
0929 20:28:24.163924 (+ 998us) server_negotiation.cc:206] Beginning negotiation
0929 20:28:24.163928 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0929 20:28:24.163941 (+ 13us) server_negotiation.cc:407] Connection header received
0929 20:28:24.163976 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:28:24.163979 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:28:24.164060 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:28:24.164155 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:28:24.165650 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:28:24.166724 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:28:24.168292 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:28:24.168937 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:28:24.169072 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:28:24.173661 (+ 4589us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:28:24.173666 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:28:24.173669 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:28:24.173697 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:28:24.176447 (+ 2750us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:28:24.177208 (+ 761us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:28:24.177213 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:28:24.177217 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:28:24.177375 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:28:24.177778 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:28:24.177783 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:28:24.177787 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:28:24.178047 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:28:24.178361 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0929 20:28:24.178650 (+ 289us) server_negotiation.cc:299] Negotiation successful
0929 20:28:24.178889 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":915,"thread_start_us":729,"threads_started":1}
I20260929 20:28:24.179311 8248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:28:24.165396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33988 (local address 127.17.53.124:32835)
0929 20:28:24.165575 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0929 20:28:24.165582 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:28:24.165601 (+ 19us) server_negotiation.cc:407] Connection header received
0929 20:28:24.165651 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:28:24.165656 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:28:24.165721 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:28:24.165789 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:28:24.166628 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:28:24.167681 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:28:24.169101 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:28:24.169767 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:28:24.169891 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:28:24.172977 (+ 3086us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:28:24.172982 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:28:24.172985 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:28:24.173015 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:28:24.176357 (+ 3342us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:28:24.177129 (+ 772us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:28:24.177137 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:28:24.177141 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:28:24.177286 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:28:24.177689 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:28:24.177695 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:28:24.177699 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:28:24.177971 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:28:24.178149 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0929 20:28:24.178504 (+ 355us) server_negotiation.cc:299] Negotiation successful
0929 20:28:24.178816 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":43,"server-negotiator.queue_time_us":67}
I20260929 20:28:24.179415 8249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:28:24.165173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.124:32835 (local address 127.0.0.1:33988)
0929 20:28:24.165350 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0929 20:28:24.165373 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0929 20:28:24.165468 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:28:24.165868 (+ 400us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:28:24.165872 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:28:24.166333 (+ 461us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:28:24.166493 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:28:24.166505 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:28:24.167818 (+ 1313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:28:24.167821 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:28:24.168959 (+ 1138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:28:24.168976 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:28:24.169939 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:28:24.169943 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:28:24.170101 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:28:24.170634 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:28:24.170649 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:28:24.172770 (+ 2121us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:28:24.176519 (+ 3749us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:28:24.176525 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:28:24.176529 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:28:24.176976 (+ 447us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:28:24.177406 (+ 430us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:28:24.177413 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:28:24.177417 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:28:24.177558 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:28:24.178102 (+ 544us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:28:24.178107 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:28:24.178260 (+ 153us) client_negotiation.cc:769] Sending connection context
0929 20:28:24.178416 (+ 156us) client_negotiation.cc:240] Negotiation successful
0929 20:28:24.179006 (+ 590us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59,"mutex_wait_us":188}
I20260929 20:28:24.185005 17621 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260929 20:28:24.185771 8245 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9faeea581fc04e18a73c4146a01ddb8a: Bootstrap starting.
I20260929 20:28:24.190181 8245 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9faeea581fc04e18a73c4146a01ddb8a: Neither blocks nor log segments found. Creating new log.
I20260929 20:28:24.191691 8245 log.cc:824] T 00000000000000000000000000000000 P 9faeea581fc04e18a73c4146a01ddb8a: Log is configured to *not* fsync() on all Append() calls
I20260929 20:28:24.188385 8072 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "0b34a238284042f29227fcb824a834b2"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46768
I20260929 20:28:24.195062 8245 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9faeea581fc04e18a73c4146a01ddb8a: No bootstrap required, opened a new log
I20260929 20:28:24.202710 8245 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9faeea581fc04e18a73c4146a01ddb8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b34a238284042f29227fcb824a834b2" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 32843 } } peers { permanent_uuid: "842df621ae7c44db813a0ed42fe76b0f" member_type: VOTER last_known_addr { host: "127.17.53.125" port: 34869 } } peers { permanent_uuid: "9faeea581fc04e18a73c4146a01ddb8a" member_type: VOTER last_known_addr { host: "127.17.53.124" port: 32835 } }
I20260929 20:28:24.203119 8245 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9faeea581fc04e18a73c4146a01ddb8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9faeea581fc04e18a73c4146a01ddb8a, State: Initialized, Role: FOLLOWER
I20260929 20:28:24.203630 8245 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9faeea581fc04e18a73c4146a01ddb8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b34a238284042f29227fcb824a834b2" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 32843 } } peers { permanent_uuid: "842df621ae7c44db813a0ed42fe76b0f" member_type: VOTER last_known_addr { host: "127.17.53.125" port: 34869 } } peers { permanent_uuid: "9faeea581fc04e18a73c4146a01ddb8a" member_type: VOTER last_known_addr { host: "127.17.53.124" port: 32835 } }
I20260929 20:28:24.204587 8253 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9faeea581fc04e18a73c4146a01ddb8a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b34a238284042f29227fcb824a834b2" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 32843 } } peers { permanent_uuid: "842df621ae7c44db813a0ed42fe76b0f" member_type: VOTER last_known_addr { host: "127.17.53.125" port: 34869 } } peers { permanent_uuid: "9faeea581fc04e18a73c4146a01ddb8a" member_type: VOTER last_known_addr { host: "127.17.53.124" port: 32835 } } }
I20260929 20:28:24.204990 8253 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9faeea581fc04e18a73c4146a01ddb8a [sys.catalog]: This master's current role is: FOLLOWER
I20260929 20:28:24.205572 8245 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9faeea581fc04e18a73c4146a01ddb8a [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:28:24.212378 8097 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.53.124" port: 32835 } attempt: 4
W20260929 20:28:24.217499 8264 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9faeea581fc04e18a73c4146a01ddb8a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 20:28:24.217691 8264 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
Sep 29 20:28:24 dist-test-slave-zbnc krb5kdc[8017](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713703, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.53.126@KRBTEST.COM for kudu/127.17.53.124@KRBTEST.COM
I20260929 20:28:24.214478 8178 master_runner.cc:186] Error getting master registration for 127.17.53.126:32843: OK, instance_id { permanent_uuid: "0b34a238284042f29227fcb824a834b2" instance_seqno: 1790713703072445 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 20:28:24.225994 8178 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 20:28:24.230587 8248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:28:24.213060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33998 (local address 127.17.53.124:32835)
0929 20:28:24.213542 (+ 482us) server_negotiation.cc:206] Beginning negotiation
0929 20:28:24.213547 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:28:24.213561 (+ 14us) server_negotiation.cc:407] Connection header received
0929 20:28:24.213606 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:28:24.213609 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:28:24.213666 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:28:24.213788 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:28:24.214825 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:28:24.216097 (+ 1272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:28:24.217948 (+ 1851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:28:24.218762 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:28:24.218943 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:28:24.223250 (+ 4307us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:28:24.223258 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:28:24.223263 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:28:24.223317 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:28:24.227350 (+ 4033us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:28:24.228252 (+ 902us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:28:24.228259 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:28:24.228263 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:28:24.228414 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:28:24.228889 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:28:24.228893 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:28:24.228896 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:28:24.229356 (+ 460us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:28:24.229511 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0929 20:28:24.229990 (+ 479us) server_negotiation.cc:299] Negotiation successful
0929 20:28:24.230224 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":388}
I20260929 20:28:24.230587 8099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:28:24.213022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.124:32835 (local address 127.0.0.1:33998)
0929 20:28:24.213264 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0929 20:28:24.213280 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0929 20:28:24.213379 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:28:24.213822 (+ 443us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:28:24.213827 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:28:24.214390 (+ 563us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:28:24.214634 (+ 244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:28:24.214650 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:28:24.216287 (+ 1637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:28:24.216304 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:28:24.217796 (+ 1492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:28:24.217815 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:28:24.218954 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:28:24.218961 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:28:24.219167 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:28:24.219929 (+ 762us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:28:24.219955 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:28:24.223002 (+ 3047us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:28:24.227518 (+ 4516us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:28:24.227523 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:28:24.227526 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:28:24.228113 (+ 587us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:28:24.228576 (+ 463us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:28:24.228581 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:28:24.228585 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:28:24.228763 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:28:24.229498 (+ 735us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:28:24.229503 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:28:24.229696 (+ 193us) client_negotiation.cc:769] Sending connection context
0929 20:28:24.229846 (+ 150us) client_negotiation.cc:240] Negotiation successful
0929 20:28:24.230159 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135}
I20260929 20:28:24.236824 8097 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b34a238284042f29227fcb824a834b2: Bootstrap starting.
I20260929 20:28:24.241180 8097 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0b34a238284042f29227fcb824a834b2: Neither blocks nor log segments found. Creating new log.
I20260929 20:28:24.242573 8097 log.cc:824] T 00000000000000000000000000000000 P 0b34a238284042f29227fcb824a834b2: Log is configured to *not* fsync() on all Append() calls
I20260929 20:28:24.246111 8097 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b34a238284042f29227fcb824a834b2: No bootstrap required, opened a new log
I20260929 20:28:24.254004 8097 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0b34a238284042f29227fcb824a834b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b34a238284042f29227fcb824a834b2" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 32843 } } peers { permanent_uuid: "842df621ae7c44db813a0ed42fe76b0f" member_type: VOTER last_known_addr { host: "127.17.53.125" port: 34869 } } peers { permanent_uuid: "9faeea581fc04e18a73c4146a01ddb8a" member_type: VOTER last_known_addr { host: "127.17.53.124" port: 32835 } }
I20260929 20:28:24.254490 8097 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0b34a238284042f29227fcb824a834b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b34a238284042f29227fcb824a834b2, State: Initialized, Role: FOLLOWER
I20260929 20:28:24.255050 8097 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0b34a238284042f29227fcb824a834b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b34a238284042f29227fcb824a834b2" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 32843 } } peers { permanent_uuid: "842df621ae7c44db813a0ed42fe76b0f" member_type: VOTER last_known_addr { host: "127.17.53.125" port: 34869 } } peers { permanent_uuid: "9faeea581fc04e18a73c4146a01ddb8a" member_type: VOTER last_known_addr { host: "127.17.53.124" port: 32835 } }
I20260929 20:28:24.256088 8267 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b34a238284042f29227fcb824a834b2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b34a238284042f29227fcb824a834b2" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 32843 } } peers { permanent_uuid: "842df621ae7c44db813a0ed42fe76b0f" member_type: VOTER last_known_addr { host: "127.17.53.125" port: 34869 } } peers { permanent_uuid: "9faeea581fc04e18a73c4146a01ddb8a" member_type: VOTER last_known_addr { host: "127.17.53.124" port: 32835 } } }
I20260929 20:28:24.256493 8267 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b34a238284042f29227fcb824a834b2 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 20:28:24.257478 8097 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0b34a238284042f29227fcb824a834b2 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:28:24.257658 8072 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0b34a238284042f29227fcb824a834b2 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260929 20:28:24.257895 8072 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0b34a238284042f29227fcb824a834b2 [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:28:24.261219 8072 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0b34a238284042f29227fcb824a834b2 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b34a238284042f29227fcb824a834b2" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 32843 } } peers { permanent_uuid: "842df621ae7c44db813a0ed42fe76b0f" member_type: VOTER last_known_addr { host: "127.17.53.125" port: 34869 } } peers { permanent_uuid: "9faeea581fc04e18a73c4146a01ddb8a" member_type: VOTER last_known_addr { host: "127.17.53.124" port: 32835 } }
I20260929 20:28:24.262550 8072 leader_election.cc:288] T 00000000000000000000000000000000 P 0b34a238284042f29227fcb824a834b2 [CANDIDATE]: Term 1 election: Requested vote from peers 842df621ae7c44db813a0ed42fe76b0f (127.17.53.125:34869), 9faeea581fc04e18a73c4146a01ddb8a (127.17.53.124:32835)
I20260929 20:28:24.264539 8220 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0b34a238284042f29227fcb824a834b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "9faeea581fc04e18a73c4146a01ddb8a"
I20260929 20:28:24.265185 8220 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9faeea581fc04e18a73c4146a01ddb8a [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:28:24.269941 8220 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 9faeea581fc04e18a73c4146a01ddb8a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0b34a238284042f29227fcb824a834b2 in term 1.
I20260929 20:28:24.271137 8047 leader_election.cc:302] T 00000000000000000000000000000000 P 0b34a238284042f29227fcb824a834b2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0b34a238284042f29227fcb824a834b2, 9faeea581fc04e18a73c4146a01ddb8a; no voters:
I20260929 20:28:24.271816 8267 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0b34a238284042f29227fcb824a834b2 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 20:28:24.285027 8267 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0b34a238284042f29227fcb824a834b2 [term 1 LEADER]: Becoming Leader. State: Replica: 0b34a238284042f29227fcb824a834b2, State: Running, Role: LEADER
I20260929 20:28:24.286795 8267 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0b34a238284042f29227fcb824a834b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b34a238284042f29227fcb824a834b2" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 32843 } } peers { permanent_uuid: "842df621ae7c44db813a0ed42fe76b0f" member_type: VOTER last_known_addr { host: "127.17.53.125" port: 34869 } } peers { permanent_uuid: "9faeea581fc04e18a73c4146a01ddb8a" member_type: VOTER last_known_addr { host: "127.17.53.124" port: 32835 } }
I20260929 20:28:24.305233 8271 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b34a238284042f29227fcb824a834b2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0b34a238284042f29227fcb824a834b2. Latest consensus state: current_term: 1 leader_uuid: "0b34a238284042f29227fcb824a834b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b34a238284042f29227fcb824a834b2" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 32843 } } peers { permanent_uuid: "842df621ae7c44db813a0ed42fe76b0f" member_type: VOTER last_known_addr { host: "127.17.53.125" port: 34869 } } peers { permanent_uuid: "9faeea581fc04e18a73c4146a01ddb8a" member_type: VOTER last_known_addr { host: "127.17.53.124" port: 32835 } } }
I20260929 20:28:24.305796 8271 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b34a238284042f29227fcb824a834b2 [sys.catalog]: This master's current role is: LEADER
W20260929 20:28:24.310801 8051 tablet.cc:2369] T 00000000000000000000000000000000 P 0b34a238284042f29227fcb824a834b2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 20:28:24.315301 8281 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260929 20:28:24.318765 8171 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.17.53.124" port: 32835 } attempt: 5
Sep 29 20:28:24 dist-test-slave-zbnc krb5kdc[8017](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790713703, etypes {rep=17 tkt=17 ses=17}, kudu/127.17.53.125@KRBTEST.COM for kudu/127.17.53.124@KRBTEST.COM
I20260929 20:28:24.339665 8174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:28:24.319512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.17.53.124:32835 (local address 127.0.0.1:34004)
0929 20:28:24.319982 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0929 20:28:24.320008 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0929 20:28:24.320159 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0929 20:28:24.321812 (+ 1653us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0929 20:28:24.321820 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0929 20:28:24.322512 (+ 692us) client_negotiation.cc:189] Negotiated authn=SASL
0929 20:28:24.322813 (+ 301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:28:24.322831 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:28:24.324575 (+ 1744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:28:24.324584 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:28:24.326546 (+ 1962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0929 20:28:24.326568 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0929 20:28:24.328261 (+ 1693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0929 20:28:24.328270 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0929 20:28:24.328606 (+ 336us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:28:24.329491 (+ 885us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0929 20:28:24.329524 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0929 20:28:24.332623 (+ 3099us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0929 20:28:24.336696 (+ 4073us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:28:24.336701 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:28:24.336705 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:28:24.337247 (+ 542us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:28:24.337681 (+ 434us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0929 20:28:24.337688 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0929 20:28:24.337695 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0929 20:28:24.337867 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0929 20:28:24.338566 (+ 699us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0929 20:28:24.338572 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0929 20:28:24.338724 (+ 152us) client_negotiation.cc:769] Sending connection context
0929 20:28:24.338950 (+ 226us) client_negotiation.cc:240] Negotiation successful
0929 20:28:24.339270 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":305}
I20260929 20:28:24.340586 8248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:28:24.320518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34004 (local address 127.17.53.124:32835)
0929 20:28:24.320747 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0929 20:28:24.320755 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 20:28:24.320778 (+ 23us) server_negotiation.cc:407] Connection header received
0929 20:28:24.320870 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:28:24.320878 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:28:24.320991 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:28:24.321110 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:28:24.323074 (+ 1964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:28:24.324361 (+ 1287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:28:24.326783 (+ 2422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:28:24.327727 (+ 944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:28:24.327893 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:28:24.332830 (+ 4937us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:28:24.332841 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:28:24.332847 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:28:24.332886 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:28:24.336522 (+ 3636us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:28:24.337382 (+ 860us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:28:24.337390 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:28:24.337394 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:28:24.337549 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:28:24.337986 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:28:24.337991 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:28:24.337994 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:28:24.338355 (+ 361us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:28:24.338504 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0929 20:28:24.339989 (+ 1485us) server_negotiation.cc:299] Negotiation successful
0929 20:28:24.340216 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
W20260929 20:28:24.344799 8199 tablet.cc:2369] T 00000000000000000000000000000000 P 9faeea581fc04e18a73c4146a01ddb8a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 20:28:24.346514 8171 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 842df621ae7c44db813a0ed42fe76b0f: Bootstrap starting.
I20260929 20:28:24.350816 8171 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 842df621ae7c44db813a0ed42fe76b0f: Neither blocks nor log segments found. Creating new log.
I20260929 20:28:24.352267 8171 log.cc:824] T 00000000000000000000000000000000 P 842df621ae7c44db813a0ed42fe76b0f: Log is configured to *not* fsync() on all Append() calls
I20260929 20:28:24.365701 8175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:28:24.316064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52460 (local address 127.17.53.125:34869)
0929 20:28:24.316249 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0929 20:28:24.316255 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:28:24.343499 (+ 27244us) server_negotiation.cc:407] Connection header received
0929 20:28:24.343718 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:28:24.343724 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:28:24.343800 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:28:24.343904 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:28:24.346755 (+ 2851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:28:24.347877 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:28:24.350093 (+ 2216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:28:24.350830 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:28:24.350957 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:28:24.357095 (+ 6138us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:28:24.357103 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:28:24.357108 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:28:24.357157 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:28:24.360951 (+ 3794us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:28:24.362357 (+ 1406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:28:24.362364 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:28:24.362379 (+ 15us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:28:24.362546 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:28:24.363426 (+ 880us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:28:24.363433 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:28:24.363438 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:28:24.363845 (+ 407us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:28:24.364105 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0929 20:28:24.365096 (+ 991us) server_negotiation.cc:299] Negotiation successful
0929 20:28:24.365346 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260929 20:28:24.366698 8171 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 842df621ae7c44db813a0ed42fe76b0f: No bootstrap required, opened a new log
I20260929 20:28:24.369915 8281 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260929 20:28:24.388540 8250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:28:24.313996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46770 (local address 127.17.53.126:32843)
0929 20:28:24.314224 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0929 20:28:24.314232 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 20:28:24.331618 (+ 17386us) server_negotiation.cc:407] Connection header received
0929 20:28:24.347102 (+ 15484us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:28:24.347112 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:28:24.347259 (+ 147us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:28:24.347346 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:28:24.351055 (+ 3709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:28:24.352439 (+ 1384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:28:24.358093 (+ 5654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:28:24.359084 (+ 991us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:28:24.359217 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:28:24.378077 (+ 18860us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:28:24.378089 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:28:24.378097 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:28:24.378157 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:28:24.384934 (+ 6777us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:28:24.385914 (+ 980us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:28:24.385921 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:28:24.385927 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:28:24.386111 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:28:24.386584 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:28:24.386589 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:28:24.386592 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:28:24.387007 (+ 415us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:28:24.387286 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0929 20:28:24.387910 (+ 624us) server_negotiation.cc:299] Negotiation successful
0929 20:28:24.388151 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260929 20:28:24.389103 8171 raft_consensus.cc:348] T 00000000000000000000000000000000 P 842df621ae7c44db813a0ed42fe76b0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b34a238284042f29227fcb824a834b2" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 32843 } } peers { permanent_uuid: "842df621ae7c44db813a0ed42fe76b0f" member_type: VOTER last_known_addr { host: "127.17.53.125" port: 34869 } } peers { permanent_uuid: "9faeea581fc04e18a73c4146a01ddb8a" member_type: VOTER last_known_addr { host: "127.17.53.124" port: 32835 } }
I20260929 20:28:24.390955 8171 raft_consensus.cc:729] T 00000000000000000000000000000000 P 842df621ae7c44db813a0ed42fe76b0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 842df621ae7c44db813a0ed42fe76b0f, State: Initialized, Role: FOLLOWER
I20260929 20:28:24.391553 8171 consensus_queue.cc:260] T 00000000000000000000000000000000 P 842df621ae7c44db813a0ed42fe76b0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b34a238284042f29227fcb824a834b2" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 32843 } } peers { permanent_uuid: "842df621ae7c44db813a0ed42fe76b0f" member_type: VOTER last_known_addr { host: "127.17.53.125" port: 34869 } } peers { permanent_uuid: "9faeea581fc04e18a73c4146a01ddb8a" member_type: VOTER last_known_addr { host: "127.17.53.124" port: 32835 } }
I20260929 20:28:24.391804 8220 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9faeea581fc04e18a73c4146a01ddb8a [term 1 FOLLOWER]: Refusing update from remote peer 0b34a238284042f29227fcb824a834b2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:28:24.393704 8271 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 0b34a238284042f29227fcb824a834b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9faeea581fc04e18a73c4146a01ddb8a" member_type: VOTER last_known_addr { host: "127.17.53.124" port: 32835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:28:24.395162 8248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:28:24.332299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34016 (local address 127.17.53.124:32835)
0929 20:28:24.340904 (+ 8605us) server_negotiation.cc:206] Beginning negotiation
0929 20:28:24.340914 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0929 20:28:24.350667 (+ 9753us) server_negotiation.cc:407] Connection header received
0929 20:28:24.354552 (+ 3885us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:28:24.354560 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:28:24.354678 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:28:24.354768 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:28:24.360417 (+ 5649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:28:24.361797 (+ 1380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:28:24.367707 (+ 5910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:28:24.368416 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:28:24.368548 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:28:24.378432 (+ 9884us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0929 20:28:24.378442 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0929 20:28:24.378449 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0929 20:28:24.378499 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0929 20:28:24.382879 (+ 4380us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:28:24.391663 (+ 8784us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:28:24.391668 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:28:24.391672 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:28:24.391866 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0929 20:28:24.392381 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0929 20:28:24.392388 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0929 20:28:24.392394 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0929 20:28:24.392794 (+ 400us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0929 20:28:24.393043 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0929 20:28:24.393619 (+ 576us) server_negotiation.cc:299] Negotiation successful
0929 20:28:24.394673 (+ 1054us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8400,"thread_start_us":11963,"threads_started":1}
I20260929 20:28:24.384640 8030 master_runner.cc:186] Error getting master registration for 127.17.53.125:34869: OK, instance_id { permanent_uuid: "842df621ae7c44db813a0ed42fe76b0f" instance_seqno: 1790713703588611 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 20:28:24.395960 8030 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260929 20:28:24.399786 8293 sys_catalog.cc:455] T 00000000000000000000000000000000 P 842df621ae7c44db813a0ed42fe76b0f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b34a238284042f29227fcb824a834b2" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 32843 } } peers { permanent_uuid: "842df621ae7c44db813a0ed42fe76b0f" member_type: VOTER last_known_addr { host: "127.17.53.125" port: 34869 } } peers { permanent_uuid: "9faeea581fc04e18a73c4146a01ddb8a" member_type: VOTER last_known_addr { host: "127.17.53.124" port: 32835 } } }
I20260929 20:28:24.400578 8293 sys_catalog.cc:458] T 00000000000000000000000000000000 P 842df621ae7c44db813a0ed42fe76b0f [sys.catalog]: This master's current role is: FOLLOWER
I20260929 20:28:24.403661 8146 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0b34a238284042f29227fcb824a834b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "842df621ae7c44db813a0ed42fe76b0f"
I20260929 20:28:24.404009 8145 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 842df621ae7c44db813a0ed42fe76b0f [term 0 FOLLOWER]: Advancing to term 1
I20260929 20:28:24.407951 8145 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 842df621ae7c44db813a0ed42fe76b0f [term 1 FOLLOWER]: Refusing update from remote peer 0b34a238284042f29227fcb824a834b2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 20:28:24.408578 8171 sys_catalog.cc:564] T 00000000000000000000000000000000 P 842df621ae7c44db813a0ed42fe76b0f [sys.catalog]: configured and running, proceeding with master startup.
I20260929 20:28:24.411494 8271 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 0b34a238284042f29227fcb824a834b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "842df621ae7c44db813a0ed42fe76b0f" member_type: VOTER last_known_addr { host: "127.17.53.125" port: 34869 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 20:28:24.414387 8146 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 842df621ae7c44db813a0ed42fe76b0f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0b34a238284042f29227fcb824a834b2 in term 1.
I20260929 20:28:24.434875 8293 sys_catalog.cc:455] T 00000000000000000000000000000000 P 842df621ae7c44db813a0ed42fe76b0f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0b34a238284042f29227fcb824a834b2. Latest consensus state: current_term: 1 leader_uuid: "0b34a238284042f29227fcb824a834b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b34a238284042f29227fcb824a834b2" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 32843 } } peers { permanent_uuid: "842df621ae7c44db813a0ed42fe76b0f" member_type: VOTER last_known_addr { host: "127.17.53.125" port: 34869 } } peers { permanent_uuid: "9faeea581fc04e18a73c4146a01ddb8a" member_type: VOTER last_known_addr { host: "127.17.53.124" port: 32835 } } }
I20260929 20:28:24.435235 8293 sys_catalog.cc:458] T 00000000000000000000000000000000 P 842df621ae7c44db813a0ed42fe76b0f [sys.catalog]: This master's current role is: FOLLOWER
I20260929 20:28:24.439051 8253 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9faeea581fc04e18a73c4146a01ddb8a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0b34a238284042f29227fcb824a834b2. Latest consensus state: current_term: 1 leader_uuid: "0b34a238284042f29227fcb824a834b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b34a238284042f29227fcb824a834b2" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 32843 } } peers { permanent_uuid: "842df621ae7c44db813a0ed42fe76b0f" member_type: VOTER last_known_addr { host: "127.17.53.125" port: 34869 } } peers { permanent_uuid: "9faeea581fc04e18a73c4146a01ddb8a" member_type: VOTER last_known_addr { host: "127.17.53.124" port: 32835 } } }
I20260929 20:28:24.439421 8253 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9faeea581fc04e18a73c4146a01ddb8a [sys.catalog]: This master's current role is: FOLLOWER
I20260929 20:28:24.444130 8292 mvcc.cc:205] Tried to move back new op lower bound from 7334763333173452800 to 7334763332777234432. Current Snapshot: MvccSnapshot[applied={T|T < 7334763333173452800}]
I20260929 20:28:24.444599 8293 sys_catalog.cc:455] T 00000000000000000000000000000000 P 842df621ae7c44db813a0ed42fe76b0f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0b34a238284042f29227fcb824a834b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b34a238284042f29227fcb824a834b2" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 32843 } } peers { permanent_uuid: "842df621ae7c44db813a0ed42fe76b0f" member_type: VOTER last_known_addr { host: "127.17.53.125" port: 34869 } } peers { permanent_uuid: "9faeea581fc04e18a73c4146a01ddb8a" member_type: VOTER last_known_addr { host: "127.17.53.124" port: 32835 } } }
I20260929 20:28:24.444846 8293 sys_catalog.cc:458] T 00000000000000000000000000000000 P 842df621ae7c44db813a0ed42fe76b0f [sys.catalog]: This master's current role is: FOLLOWER
I20260929 20:28:24.462210 8294 mvcc.cc:205] Tried to move back new op lower bound from 7334763333173452800 to 7334763332777234432. Current Snapshot: MvccSnapshot[applied={T|T < 7334763333173452800}]
I20260929 20:28:24.463680 8253 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9faeea581fc04e18a73c4146a01ddb8a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0b34a238284042f29227fcb824a834b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b34a238284042f29227fcb824a834b2" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 32843 } } peers { permanent_uuid: "842df621ae7c44db813a0ed42fe76b0f" member_type: VOTER last_known_addr { host: "127.17.53.125" port: 34869 } } peers { permanent_uuid: "9faeea581fc04e18a73c4146a01ddb8a" member_type: VOTER last_known_addr { host: "127.17.53.124" port: 32835 } } }
I20260929 20:28:24.463986 8253 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9faeea581fc04e18a73c4146a01ddb8a [sys.catalog]: This master's current role is: FOLLOWER
I20260929 20:28:24.468953 8267 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b34a238284042f29227fcb824a834b2 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0b34a238284042f29227fcb824a834b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b34a238284042f29227fcb824a834b2" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 32843 } } peers { permanent_uuid: "842df621ae7c44db813a0ed42fe76b0f" member_type: VOTER last_known_addr { host: "127.17.53.125" port: 34869 } } peers { permanent_uuid: "9faeea581fc04e18a73c4146a01ddb8a" member_type: VOTER last_known_addr { host: "127.17.53.124" port: 32835 } } }
I20260929 20:28:24.469321 8267 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b34a238284042f29227fcb824a834b2 [sys.catalog]: This master's current role is: LEADER
I20260929 20:28:24.469574 8267 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b34a238284042f29227fcb824a834b2 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0b34a238284042f29227fcb824a834b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0b34a238284042f29227fcb824a834b2" member_type: VOTER last_known_addr { host: "127.17.53.126" port: 32843 } } peers { permanent_uuid: "842df621ae7c44db813a0ed42fe76b0f" member_type: VOTER last_known_addr { host: "127.17.53.125" port: 34869 } } peers { permanent_uuid: "9faeea581fc04e18a73c4146a01ddb8a" member_type: VOTER last_known_addr { host: "127.17.53.124" port: 32835 } } }
I20260929 20:28:24.469839 8267 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b34a238284042f29227fcb824a834b2 [sys.catalog]: This master's current role is: LEADER
I20260929 20:28:24.471287 8281 catalog_manager.cc:1347] Generated new cluster ID: ac88efefba1f46f79c49ffa978e52bd5
I20260929 20:28:24.471390 8281 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260929 20:28:24.497421 8281 catalog_manager.cc:1370] Generated new certificate authority record
I20260929 20:28:24.499490 8281 catalog_manager.cc:1504] Loading token signing keys...
I20260929 20:28:24.536347 8281 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0b34a238284042f29227fcb824a834b2: Generated new TSK 0
I20260929 20:28:24.537716 8281 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260929 20:28:24.569253 8311 catalog_manager.cc:1259] Loaded cluster ID: ac88efefba1f46f79c49ffa978e52bd5
I20260929 20:28:24.569469 8311 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 842df621ae7c44db813a0ed42fe76b0f: loading cluster ID for follower catalog manager: success
I20260929 20:28:24.573511 8311 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 842df621ae7c44db813a0ed42fe76b0f: acquiring CA information for follower catalog manager: success
I20260929 20:28:24.576370 8311 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 842df621ae7c44db813a0ed42fe76b0f: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260929 20:28:24.685927 8250 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:28:24.663785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46786 (local address 127.17.53.126:32843)
0929 20:28:24.663998 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0929 20:28:24.664004 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:28:24.681498 (+ 17494us) server_negotiation.cc:407] Connection header received
0929 20:28:24.684232 (+ 2734us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:28:24.684239 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:28:24.684340 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:28:24.684422 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:28:24.685626 (+ 1204us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46786: BlockingRecv error: recv got EOF from 127.0.0.1:46786 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
W20260929 20:28:24.690106 8248 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:28:24.682152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34026 (local address 127.17.53.124:32835)
0929 20:28:24.682360 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0929 20:28:24.682367 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:28:24.688479 (+ 6112us) server_negotiation.cc:407] Connection header received
0929 20:28:24.688718 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:28:24.688724 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:28:24.688825 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:28:24.688897 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:28:24.689854 (+ 957us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34026: BlockingRecv error: recv got EOF from 127.0.0.1:34026 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
W20260929 20:28:24.692200 8175 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:28:24.665715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52470 (local address 127.17.53.125:34869)
0929 20:28:24.665951 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0929 20:28:24.665957 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:28:24.687895 (+ 21938us) server_negotiation.cc:407] Connection header received
0929 20:28:24.688044 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:28:24.688050 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:28:24.688161 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:28:24.688279 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:28:24.691947 (+ 3668us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52470: BlockingRecv error: recv got EOF from 127.0.0.1:52470 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
I20260929 20:28:24.739377 8175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:28:24.720255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52478 (local address 127.17.53.125:34869)
0929 20:28:24.720444 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0929 20:28:24.720451 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:28:24.731095 (+ 10644us) server_negotiation.cc:407] Connection header received
0929 20:28:24.731670 (+ 575us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:28:24.731689 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:28:24.731907 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:28:24.732044 (+ 137us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 20:28:24.732843 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:28:24.734001 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:28:24.736088 (+ 2087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:28:24.736812 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:28:24.736915 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:28:24.737737 (+ 822us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 20:28:24.738469 (+ 732us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 20:28:24.738598 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0929 20:28:24.738926 (+ 328us) server_negotiation.cc:299] Negotiation successful
0929 20:28:24.739026 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
W20260929 20:28:24.788489 8248 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:28:24.776164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34028 (local address 127.17.53.124:32835)
0929 20:28:24.776420 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0929 20:28:24.776428 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 20:28:24.786167 (+ 9739us) server_negotiation.cc:407] Connection header received
0929 20:28:24.786807 (+ 640us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:28:24.786816 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:28:24.786975 (+ 159us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:28:24.787064 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:28:24.788213 (+ 1149us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34028: BlockingRecv error: recv got EOF from 127.0.0.1:34028 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
I20260929 20:28:24.841351 8175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:28:24.823327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52480 (local address 127.17.53.125:34869)
0929 20:28:24.823524 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0929 20:28:24.823530 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:28:24.834157 (+ 10627us) server_negotiation.cc:407] Connection header received
0929 20:28:24.834687 (+ 530us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:28:24.834695 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:28:24.834789 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:28:24.834875 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 20:28:24.835586 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:28:24.836812 (+ 1226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:28:24.839097 (+ 2285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:28:24.839781 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:28:24.839884 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:28:24.840497 (+ 613us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 20:28:24.840658 (+ 161us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 20:28:24.840744 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0929 20:28:24.840962 (+ 218us) server_negotiation.cc:299] Negotiation successful
0929 20:28:24.841062 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
W20260929 20:28:24.887503 8248 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:28:24.875437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34036 (local address 127.17.53.124:32835)
0929 20:28:24.875666 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0929 20:28:24.875683 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0929 20:28:24.885470 (+ 9787us) server_negotiation.cc:407] Connection header received
0929 20:28:24.885992 (+ 522us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:28:24.885998 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:28:24.886098 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:28:24.886175 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:28:24.887248 (+ 1073us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34036: BlockingRecv error: recv got EOF from 127.0.0.1:34036 (error 108)
Metrics: {"server-negotiator.queue_time_us":101}
I20260929 20:28:24.943167 8175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:28:24.923170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52488 (local address 127.17.53.125:34869)
0929 20:28:24.923360 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0929 20:28:24.923366 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:28:24.935932 (+ 12566us) server_negotiation.cc:407] Connection header received
0929 20:28:24.936470 (+ 538us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:28:24.936478 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:28:24.936574 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:28:24.936642 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 20:28:24.937431 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:28:24.938645 (+ 1214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:28:24.940802 (+ 2157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:28:24.941502 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:28:24.941604 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:28:24.942229 (+ 625us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 20:28:24.942392 (+ 163us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 20:28:24.942451 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0929 20:28:24.942716 (+ 265us) server_negotiation.cc:299] Negotiation successful
0929 20:28:24.942804 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
W20260929 20:28:24.952924 8248 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:28:24.950301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34042 (local address 127.17.53.124:32835)
0929 20:28:24.950500 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0929 20:28:24.950505 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:28:24.951095 (+ 590us) server_negotiation.cc:407] Connection header received
0929 20:28:24.951377 (+ 282us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:28:24.951383 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:28:24.951487 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:28:24.951557 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:28:24.952612 (+ 1055us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34042: BlockingRecv error: recv got EOF from 127.0.0.1:34042 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
I20260929 20:28:24.978149 8175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:28:24.966435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52490 (local address 127.17.53.125:34869)
0929 20:28:24.966679 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0929 20:28:24.966686 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:28:24.971485 (+ 4799us) server_negotiation.cc:407] Connection header received
0929 20:28:24.971788 (+ 303us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:28:24.971795 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:28:24.971924 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:28:24.971988 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 20:28:24.972651 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:28:24.973784 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:28:24.975992 (+ 2208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:28:24.976688 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:28:24.976809 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:28:24.977323 (+ 514us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 20:28:24.977470 (+ 147us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 20:28:24.977571 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0929 20:28:24.977763 (+ 192us) server_negotiation.cc:299] Negotiation successful
0929 20:28:24.977850 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114}
W20260929 20:28:24.987483 8248 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:28:24.984902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34054 (local address 127.17.53.124:32835)
0929 20:28:24.985162 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0929 20:28:24.985168 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:28:24.985690 (+ 522us) server_negotiation.cc:407] Connection header received
0929 20:28:24.985978 (+ 288us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:28:24.985986 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:28:24.986086 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:28:24.986161 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:28:24.987257 (+ 1096us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34054: BlockingRecv error: recv got EOF from 127.0.0.1:34054 (error 108)
Metrics: {"server-negotiator.queue_time_us":148}
I20260929 20:28:25.007740 8175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:28:24.999861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52502 (local address 127.17.53.125:34869)
0929 20:28:25.000057 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0929 20:28:25.000063 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:28:25.000619 (+ 556us) server_negotiation.cc:407] Connection header received
0929 20:28:25.000964 (+ 345us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:28:25.000970 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:28:25.001064 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:28:25.001142 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 20:28:25.001808 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:28:25.003047 (+ 1239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:28:25.005145 (+ 2098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:28:25.005998 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:28:25.006137 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:28:25.006688 (+ 551us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 20:28:25.006957 (+ 269us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 20:28:25.007062 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0929 20:28:25.007269 (+ 207us) server_negotiation.cc:299] Negotiation successful
0929 20:28:25.007359 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
W20260929 20:28:25.016991 8248 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:28:25.014944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34056 (local address 127.17.53.124:32835)
0929 20:28:25.015160 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0929 20:28:25.015165 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:28:25.015481 (+ 316us) server_negotiation.cc:407] Connection header received
0929 20:28:25.015681 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:28:25.015685 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:28:25.015780 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:28:25.015863 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:28:25.016760 (+ 897us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34056: BlockingRecv error: recv got EOF from 127.0.0.1:34056 (error 108)
Metrics: {"server-negotiator.queue_time_us":87}
I20260929 20:28:25.043195 8175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:28:25.036728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52510 (local address 127.17.53.125:34869)
0929 20:28:25.036918 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0929 20:28:25.036925 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:28:25.037437 (+ 512us) server_negotiation.cc:407] Connection header received
0929 20:28:25.037682 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:28:25.037687 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:28:25.037793 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:28:25.037886 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 20:28:25.038338 (+ 452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:28:25.039241 (+ 903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:28:25.040878 (+ 1637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:28:25.041308 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:28:25.041671 (+ 363us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:28:25.042305 (+ 634us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 20:28:25.042424 (+ 119us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 20:28:25.042561 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0929 20:28:25.042751 (+ 190us) server_negotiation.cc:299] Negotiation successful
0929 20:28:25.042885 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
W20260929 20:28:25.051852 8248 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:28:25.049435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34068 (local address 127.17.53.124:32835)
0929 20:28:25.049649 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0929 20:28:25.049655 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:28:25.050006 (+ 351us) server_negotiation.cc:407] Connection header received
0929 20:28:25.050185 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:28:25.050189 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:28:25.050287 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:28:25.050371 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:28:25.051557 (+ 1186us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34068: BlockingRecv error: recv got EOF from 127.0.0.1:34068 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
I20260929 20:28:25.096793 8175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:28:25.089479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52526 (local address 127.17.53.125:34869)
0929 20:28:25.089738 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0929 20:28:25.089744 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0929 20:28:25.090095 (+ 351us) server_negotiation.cc:407] Connection header received
0929 20:28:25.090332 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:28:25.090340 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:28:25.090451 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:28:25.090552 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 20:28:25.091117 (+ 565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:28:25.092290 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:28:25.094361 (+ 2071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:28:25.095138 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:28:25.095262 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:28:25.095848 (+ 586us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 20:28:25.096019 (+ 171us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 20:28:25.096125 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0929 20:28:25.096325 (+ 200us) server_negotiation.cc:299] Negotiation successful
0929 20:28:25.096399 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
W20260929 20:28:25.105834 8248 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:28:25.103569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34076 (local address 127.17.53.124:32835)
0929 20:28:25.103786 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0929 20:28:25.103791 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:28:25.104158 (+ 367us) server_negotiation.cc:407] Connection header received
0929 20:28:25.104318 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:28:25.104322 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:28:25.104396 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:28:25.104477 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:28:25.105475 (+ 998us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34076: BlockingRecv error: recv got EOF from 127.0.0.1:34076 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
I20260929 20:28:25.178345 8175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:28:25.173320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52534 (local address 127.17.53.125:34869)
0929 20:28:25.173502 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0929 20:28:25.173507 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0929 20:28:25.173889 (+ 382us) server_negotiation.cc:407] Connection header received
0929 20:28:25.174074 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:28:25.174078 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:28:25.174148 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:28:25.174242 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 20:28:25.174651 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:28:25.175491 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:28:25.176873 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:28:25.177286 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:28:25.177382 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:28:25.177720 (+ 338us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 20:28:25.177827 (+ 107us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 20:28:25.177898 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0929 20:28:25.178049 (+ 151us) server_negotiation.cc:299] Negotiation successful
0929 20:28:25.178111 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
W20260929 20:28:25.186002 8248 negotiation.cc:337] Failed RPC negotiation. Trace:
0929 20:28:25.184151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34080 (local address 127.17.53.124:32835)
0929 20:28:25.184393 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0929 20:28:25.184400 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0929 20:28:25.184685 (+ 285us) server_negotiation.cc:407] Connection header received
0929 20:28:25.184822 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:28:25.184826 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:28:25.184911 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:28:25.184998 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0929 20:28:25.185780 (+ 782us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34080: BlockingRecv error: recv got EOF from 127.0.0.1:34080 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
I20260929 20:28:25.219704 8264 catalog_manager.cc:1259] Loaded cluster ID: ac88efefba1f46f79c49ffa978e52bd5
I20260929 20:28:25.219820 8264 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 9faeea581fc04e18a73c4146a01ddb8a: loading cluster ID for follower catalog manager: success
I20260929 20:28:25.222332 8264 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 9faeea581fc04e18a73c4146a01ddb8a: acquiring CA information for follower catalog manager: success
I20260929 20:28:25.224170 8264 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 9faeea581fc04e18a73c4146a01ddb8a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 20:28:25.323345 8175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:28:25.317792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52538 (local address 127.17.53.125:34869)
0929 20:28:25.318120 (+ 328us) server_negotiation.cc:206] Beginning negotiation
0929 20:28:25.318128 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 20:28:25.318422 (+ 294us) server_negotiation.cc:407] Connection header received
0929 20:28:25.318677 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:28:25.318681 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:28:25.318766 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:28:25.318925 (+ 159us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 20:28:25.319368 (+ 443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:28:25.320102 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:28:25.321529 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:28:25.321949 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:28:25.322095 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:28:25.322544 (+ 449us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 20:28:25.322662 (+ 118us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 20:28:25.322776 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0929 20:28:25.322980 (+ 204us) server_negotiation.cc:299] Negotiation successful
0929 20:28:25.323066 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
I20260929 20:28:25.334805 8248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:28:25.329175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34092 (local address 127.17.53.124:32835)
0929 20:28:25.329440 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0929 20:28:25.329448 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0929 20:28:25.329634 (+ 186us) server_negotiation.cc:407] Connection header received
0929 20:28:25.329779 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:28:25.329783 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:28:25.329942 (+ 159us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:28:25.330041 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 20:28:25.330430 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:28:25.331205 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:28:25.332553 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:28:25.332965 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:28:25.333114 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:28:25.333632 (+ 518us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 20:28:25.334249 (+ 617us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 20:28:25.334423 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0929 20:28:25.334508 (+ 85us) server_negotiation.cc:299] Negotiation successful
0929 20:28:25.334577 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260929 20:28:25.348419 8175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:28:25.340810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52552 (local address 127.17.53.125:34869)
0929 20:28:25.341031 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0929 20:28:25.341040 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 20:28:25.342122 (+ 1082us) server_negotiation.cc:407] Connection header received
0929 20:28:25.342386 (+ 264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:28:25.342391 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:28:25.342473 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:28:25.342540 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 20:28:25.343527 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:28:25.344535 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:28:25.346337 (+ 1802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:28:25.346967 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:28:25.347095 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:28:25.347643 (+ 548us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 20:28:25.347794 (+ 151us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 20:28:25.347851 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0929 20:28:25.348068 (+ 217us) server_negotiation.cc:299] Negotiation successful
0929 20:28:25.348152 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260929 20:28:25.349615 8248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:28:25.342279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34096 (local address 127.17.53.124:32835)
0929 20:28:25.342517 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0929 20:28:25.342526 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 20:28:25.343139 (+ 613us) server_negotiation.cc:407] Connection header received
0929 20:28:25.343357 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:28:25.343362 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:28:25.343434 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:28:25.343540 (+ 106us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 20:28:25.344106 (+ 566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:28:25.345128 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:28:25.347285 (+ 2157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:28:25.347991 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:28:25.348104 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:28:25.348826 (+ 722us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 20:28:25.348986 (+ 160us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 20:28:25.349083 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0929 20:28:25.349273 (+ 190us) server_negotiation.cc:299] Negotiation successful
0929 20:28:25.349354 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260929 20:28:25.350059 8417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0929 20:28:25.340692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46796 (local address 127.17.53.126:32843)
0929 20:28:25.341467 (+ 775us) server_negotiation.cc:206] Beginning negotiation
0929 20:28:25.341476 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0929 20:28:25.341904 (+ 428us) server_negotiation.cc:407] Connection header received
0929 20:28:25.342213 (+ 309us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0929 20:28:25.342218 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0929 20:28:25.342479 (+ 261us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0929 20:28:25.342650 (+ 171us) server_negotiation.cc:226] Negotiated authn=TOKEN
0929 20:28:25.343576 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:28:25.344526 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:28:25.346710 (+ 2184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0929 20:28:25.347433 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0929 20:28:25.347547 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0929 20:28:25.348266 (+ 719us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0929 20:28:25.349031 (+ 765us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0929 20:28:25.349183 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0929 20:28:25.349623 (+ 440us) server_negotiation.cc:299] Negotiation successful
0929 20:28:25.349742 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":624,"thread_start_us":484,"threads_started":1}
I20260929 20:28:25.354534 17621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu pid 8030
I20260929 20:28:25.355692 8092 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:28:25.467695 17621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu with pid 8030
I20260929 20:28:25.489410 17621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu pid 8104
I20260929 20:28:25.490621 8166 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:28:25.602582 17621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu with pid 8104
I20260929 20:28:25.623735 17621 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu pid 8178
I20260929 20:28:25.624925 8240 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 20:28:25.731628 17621 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfEhy_t/build/asan/bin/kudu with pid 8178
2026-09-29T20:28:25Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (4539 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (8977 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (366536 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST